[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:56 dist-test-slave-45dx krb5kdc[4409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:56 dist-test-slave-45dx krb5kdc[4409](info): set up 2 sockets
Aug 10 06:24:56 dist-test-slave-45dx krb5kdc[4409](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 06:24:57.914088 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.009s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:24:57 dist-test-slave-45dx krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:24:58.041635 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46421
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:46421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:58.164053 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:58.164417 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:58.164521 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:58.169308 4432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:24:58.169394 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:58.169425 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:58.169449 4432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:24:58.169515 4432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:24:58.175859 4432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:46421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:58.178108 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:58.179383 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:24:58.194865 4432 server_base.cc:1034] running on GCE node
W20260810 06:24:58.194908 4441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.194882 4446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.194892 4442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:58.195799 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:58.197500 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:58.198698 4432 hybrid_clock.cc:648] HybridClock initialized: now 1786343098198647 us; error 66 us; skew 500 ppm
Aug 10 06:24:58 dist-test-slave-45dx krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:58.201774 4432 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:24:58.202916 4432 webserver.cc:460] Webserver started at http://127.4.73.62:40781/ using document root <none> and password file <none>
I20260810 06:24:58.203809 4432 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:58.203915 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:58.204350 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:58.206940 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "af92fb5a874846d88deda490c75fb965"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "7273f1702fc33443905ef3377281320b"
server_key_iv: "db6a185eb7578d3cf69743ff2290c8c2"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.207695 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "af92fb5a874846d88deda490c75fb965"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "7273f1702fc33443905ef3377281320b"
server_key_iv: "db6a185eb7578d3cf69743ff2290c8c2"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.212500 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260810 06:24:58.216593 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.218077 4432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:24:58.218292 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "af92fb5a874846d88deda490c75fb965"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "7273f1702fc33443905ef3377281320b"
server_key_iv: "db6a185eb7578d3cf69743ff2290c8c2"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.218415 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:58.259509 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:58.263543 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:58.263801 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:58.274730 4432 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:46421
I20260810 06:24:58.275132 4538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:46421 every 8 connection(s)
I20260810 06:24:58.276075 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:24:58.279681 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 4432
I20260810 06:24:58.279901 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:24:58.281862 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:24:58.282238 4388 external_mini_cluster.cc:1428] Setting key 5859db5a05e91e69ba74d91d58ab1821
I20260810 06:24:58.292717 4539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965: Bootstrap starting.
I20260810 06:24:58.296250 4539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965: Neither blocks nor log segments found. Creating new log.
I20260810 06:24:58.297423 4539 log.cc:824] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965: Log is configured to *not* fsync() on all Append() calls
I20260810 06:24:58.299932 4539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965: No bootstrap required, opened a new log
Aug 10 06:24:58 dist-test-slave-45dx krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:24:58.304670 4539 raft_consensus.cc:348] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af92fb5a874846d88deda490c75fb965" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46421 } }
I20260810 06:24:58.304965 4539 raft_consensus.cc:374] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:24:58.305074 4539 raft_consensus.cc:729] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af92fb5a874846d88deda490c75fb965, State: Initialized, Role: FOLLOWER
I20260810 06:24:58.305773 4539 consensus_queue.cc:260] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af92fb5a874846d88deda490c75fb965" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46421 } }
I20260810 06:24:58.305998 4539 raft_consensus.cc:388] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:24:58.306147 4539 raft_consensus.cc:482] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:24:58.306305 4539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:24:58.307454 4539 raft_consensus.cc:504] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af92fb5a874846d88deda490c75fb965" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46421 } }
I20260810 06:24:58.308112 4539 leader_election.cc:302] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af92fb5a874846d88deda490c75fb965; no voters:
I20260810 06:24:58.308595 4539 leader_election.cc:288] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:24:58.308893 4547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:24:58.309152 4547 raft_consensus.cc:686] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [term 1 LEADER]: Becoming Leader. State: Replica: af92fb5a874846d88deda490c75fb965, State: Running, Role: LEADER
I20260810 06:24:58.310046 4547 consensus_queue.cc:237] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af92fb5a874846d88deda490c75fb965" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46421 } }
I20260810 06:24:58.310176 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:58.291018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.4.73.62:46421)
0810 06:24:58.291525 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:58.291542 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:58.291632 (+ 90us) server_negotiation.cc:407] Connection header received
0810 06:24:58.293390 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:58.293406 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:58.293767 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:58.294084 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:58.296387 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.297586 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.299034 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.299488 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.299652 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:58.304007 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:58.304038 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:58.304050 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:58.304078 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:58.305903 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:58.306647 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:58.306651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:58.306655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:58.306763 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:58.307174 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:58.307176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:58.307177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:58.307694 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:58.307857 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:58.308710 (+ 853us) server_negotiation.cc:299] Negotiation successful
0810 06:24:58.309044 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":154,"threads_started":1}
I20260810 06:24:58.310834 4539 sys_catalog.cc:564] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:24:58.315173 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af92fb5a874846d88deda490c75fb965. Latest consensus state: current_term: 1 leader_uuid: "af92fb5a874846d88deda490c75fb965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af92fb5a874846d88deda490c75fb965" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46421 } } }
I20260810 06:24:58.315301 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [sys.catalog]: This master's current role is: LEADER
I20260810 06:24:58.315636 4549 sys_catalog.cc:455] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af92fb5a874846d88deda490c75fb965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af92fb5a874846d88deda490c75fb965" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46421 } } }
I20260810 06:24:58.315733 4549 sys_catalog.cc:458] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965 [sys.catalog]: This master's current role is: LEADER
W20260810 06:24:58.318354 4573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:24:58.318500 4573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:24:58.318588 4575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:24:58.319681 4575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:24:58.327507 4575 catalog_manager.cc:1347] Generated new cluster ID: 9531537303ae4aa0837bca14a2fd0f5a
I20260810 06:24:58.327587 4575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:24:58.346721 4575 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:24:58.348124 4575 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:24:58.358781 4575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af92fb5a874846d88deda490c75fb965: Generated new TSK 0
I20260810 06:24:58.359674 4575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:24:58.391126 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:46421
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:58.509313 4585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:58.509593 4585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:58.509668 4585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:58.513833 4585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:58.513906 4585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:58.513993 4585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:24:58.519168 4585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:46421
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:58.520210 4585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:58.521196 4585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:58.531114 4600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.531150 4596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.531107 4597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:58.531345 4585 server_base.cc:1034] running on GCE node
I20260810 06:24:58.531772 4585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:58.532377 4585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:58.533550 4585 hybrid_clock.cc:648] HybridClock initialized: now 1786343098533538 us; error 29 us; skew 500 ppm
Aug 10 06:24:58 dist-test-slave-45dx krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:58.536235 4585 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:24:58.537061 4585 webserver.cc:460] Webserver started at http://127.4.73.1:42003/ using document root <none> and password file <none>
I20260810 06:24:58.537611 4585 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:58.537698 4585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:58.537933 4585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:58.539903 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "b072507a3e00490a93e50042496813df"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "f0f2c84903798d0563f95452d4e4cfed"
server_key_iv: "ba1c311b77a61e8932a641b505fe4dc6"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.540513 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "b072507a3e00490a93e50042496813df"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "f0f2c84903798d0563f95452d4e4cfed"
server_key_iv: "ba1c311b77a61e8932a641b505fe4dc6"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.544093 4585 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:24:58.546360 4609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.547418 4585 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:58.547542 4585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "b072507a3e00490a93e50042496813df"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "f0f2c84903798d0563f95452d4e4cfed"
server_key_iv: "ba1c311b77a61e8932a641b505fe4dc6"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.547658 4585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:58.558919 4585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:58.562062 4585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:58.562211 4585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:58.562804 4585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:58.563889 4585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:58.563961 4585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.564000 4585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:58.564021 4585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.575348 4585 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:42735
I20260810 06:24:58.575376 4798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:42735 every 8 connection(s)
I20260810 06:24:58.576190 4585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:24:58.578254 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 4585
I20260810 06:24:58.578378 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:24:58.578644 4388 external_mini_cluster.cc:1428] Setting key dad8e2632953a72f49d37e78fecee5c7
Aug 10 06:24:58 dist-test-slave-45dx krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:24:58.591632 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:58.579126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:45311 (local address 127.4.73.62:46421)
0810 06:24:58.579284 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:58.579289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:58.580319 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 06:24:58.581951 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:58.581957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:58.582036 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:58.582154 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:58.583254 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.583801 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.585049 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.585295 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.585408 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:58.587905 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:58.587908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:58.587911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:58.587935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:58.589513 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:58.590072 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:58.590075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:58.590077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:58.590160 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:58.590555 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:58.590558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:58.590560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:58.590745 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:58.590854 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:58.591329 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 06:24:58.591467 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:24:58.592624 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:58.579540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:46421 (local address 127.4.73.1:45311)
0810 06:24:58.580145 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:58.580198 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:58.581696 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:58.582278 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:58.582294 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:58.582633 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:58.583093 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:58.583118 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.583937 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.583941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:58.584922 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:58.584932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.585443 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.585447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:58.585615 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:58.586256 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:58.586272 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:58.587738 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:58.589636 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:58.589642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:58.589656 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:58.589968 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:58.590254 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:58.590257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:58.590259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:58.590405 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:58.590854 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:58.590862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:58.591097 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 06:24:58.591282 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 06:24:58.591686 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":133,"threads_started":1}
I20260810 06:24:58.593814 4799 heartbeater.cc:347] Connected to a master server at 127.4.73.62:46421
I20260810 06:24:58.594038 4799 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:58.594635 4799 heartbeater.cc:511] Master 127.4.73.62:46421 requested a full tablet report, sending...
I20260810 06:24:58.596840 4471 ts_manager.cc:194] Registered new tserver with Master: b072507a3e00490a93e50042496813df (127.4.73.1:42735)
I20260810 06:24:58.598244 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:45311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:24:58.608893 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:46421
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:58.725804 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:58.726046 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:58.726128 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:58.730044 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:58.730095 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:58.730182 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:24:58.735221 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:46421
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:58.736229 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:58.737267 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:58.746892 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.746942 4824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.746942 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:58.747154 4809 server_base.cc:1034] running on GCE node
I20260810 06:24:58.747612 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:58.748214 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:58.749459 4809 hybrid_clock.cc:648] HybridClock initialized: now 1786343098749445 us; error 30 us; skew 500 ppm
Aug 10 06:24:58 dist-test-slave-45dx krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:58.752280 4809 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:24:58.753100 4809 webserver.cc:460] Webserver started at http://127.4.73.2:38085/ using document root <none> and password file <none>
I20260810 06:24:58.753587 4809 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:58.753645 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:58.753811 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:58.755733 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "194f27958ed34e809f30c7945d3152d2"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "4bf35a7441817bf85f2bc543e4d85bc8"
server_key_iv: "08662a75fa3c81b34ebaf52deb916bf6"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.756253 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "194f27958ed34e809f30c7945d3152d2"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "4bf35a7441817bf85f2bc543e4d85bc8"
server_key_iv: "08662a75fa3c81b34ebaf52deb916bf6"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.760054 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 06:24:58.762941 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.764439 4809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 06:24:58.764550 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "194f27958ed34e809f30c7945d3152d2"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "4bf35a7441817bf85f2bc543e4d85bc8"
server_key_iv: "08662a75fa3c81b34ebaf52deb916bf6"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.764654 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:58.783429 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:58.786425 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:58.786584 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:58.787145 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:58.787923 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:58.787971 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.788012 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:58.788033 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.798784 4809 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:34173
I20260810 06:24:58.798861 5026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:34173 every 8 connection(s)
I20260810 06:24:58.799654 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:24:58.806285 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 4809
I20260810 06:24:58.806386 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:24:58.806679 4388 external_mini_cluster.cc:1428] Setting key 61d9705e6bab51d27501ef69cef271e2
Aug 10 06:24:58 dist-test-slave-45dx krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:24:58.814246 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:58.801535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:34021 (local address 127.4.73.62:46421)
0810 06:24:58.801667 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:58.801676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:58.802529 (+ 853us) server_negotiation.cc:407] Connection header received
0810 06:24:58.803872 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:58.803876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:58.803932 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:58.804041 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:58.805251 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.805789 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.807108 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.807350 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.807480 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:58.810126 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:58.810133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:58.810139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:58.810180 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:58.811787 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:58.812385 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:58.812391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:58.812396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:58.812497 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:58.812906 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:58.812912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:58.812917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:58.813192 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:58.813292 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:58.813851 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 06:24:58.814011 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:24:58.815267 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:58.801907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:46421 (local address 127.4.73.2:34021)
0810 06:24:58.802379 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:58.802421 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:58.803663 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:58.804207 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:58.804214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:58.804617 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:58.805094 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:58.805108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.805927 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.805931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:58.806973 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:58.806983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:58.807514 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:58.807518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:58.807698 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:58.808370 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:58.808387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:58.809935 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:58.811915 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:58.811922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:58.811938 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:58.812227 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:58.812600 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:58.812603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:58.812604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:58.812796 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:58.813307 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:58.813313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:58.813558 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 06:24:58.813745 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 06:24:58.814103 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":155,"threads_started":1}
I20260810 06:24:58.816569 5028 heartbeater.cc:347] Connected to a master server at 127.4.73.62:46421
I20260810 06:24:58.816839 5028 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:58.817315 5028 heartbeater.cc:511] Master 127.4.73.62:46421 requested a full tablet report, sending...
I20260810 06:24:58.818368 4471 ts_manager.cc:194] Registered new tserver with Master: 194f27958ed34e809f30c7945d3152d2 (127.4.73.2:34173)
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:24:58.818954 4471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:34021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:24:58.839701 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:46421
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:24:58.957023 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:24:58.957319 5035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:24:58.957394 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:24:58.961421 5035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:24:58.961508 5035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:24:58.961603 5035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:24:58.967051 5035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:46421
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:24:58.968158 5035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:24:58.969430 5035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:24:58.978924 5048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.978952 5053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:24:58.978925 5047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:24:58.979314 5035 server_base.cc:1034] running on GCE node
I20260810 06:24:58.979704 5035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:24:58.980286 5035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:24:58.981532 5035 hybrid_clock.cc:648] HybridClock initialized: now 1786343098981521 us; error 33 us; skew 500 ppm
Aug 10 06:24:58 dist-test-slave-45dx krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:24:58.984223 5035 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:24:58.985105 5035 webserver.cc:460] Webserver started at http://127.4.73.3:43885/ using document root <none> and password file <none>
I20260810 06:24:58.985657 5035 fs_manager.cc:359] Metadata directory not provided
I20260810 06:24:58.985750 5035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:24:58.985980 5035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:24:58.988090 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "9799a8e856a94dbfaf2f26810b948630"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "85754e32e9d4cb253955139a8f53c85b"
server_key_iv: "beb16187940c930ba6d6533fd3e947eb"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.988760 5035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "9799a8e856a94dbfaf2f26810b948630"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "85754e32e9d4cb253955139a8f53c85b"
server_key_iv: "beb16187940c930ba6d6533fd3e947eb"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.992401 5035 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:24:58.994689 5068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:58.995823 5035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:24:58.995955 5035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "9799a8e856a94dbfaf2f26810b948630"
format_stamp: "Formatted at 2026-08-10 06:24:58 on dist-test-slave-45dx"
server_key: "85754e32e9d4cb253955139a8f53c85b"
server_key_iv: "beb16187940c930ba6d6533fd3e947eb"
server_key_version: "encryptionkey@0"
I20260810 06:24:58.996063 5035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:24:59.015326 5035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:24:59.018939 5035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:24:59.019177 5035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:24:59.019804 5035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:24:59.020682 5035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:24:59.020753 5035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:59.020829 5035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:24:59.020869 5035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:24:59.033210 5035 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:38081
I20260810 06:24:59.033238 5267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:38081 every 8 connection(s)
I20260810 06:24:59.034138 5035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:24:59.037868 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5035
I20260810 06:24:59.038000 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:24:59.038275 4388 external_mini_cluster.cc:1428] Setting key af5f6418c3fee10f137f39b0a579e271
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:24:59.049015 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:59.036233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:44017 (local address 127.4.73.62:46421)
0810 06:24:59.036445 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:59.036449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:59.037431 (+ 982us) server_negotiation.cc:407] Connection header received
0810 06:24:59.038935 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:59.038939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:59.038995 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:59.039085 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:59.040632 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.041303 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.042472 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.042689 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.042788 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:59.045290 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:59.045294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:59.045296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:59.045323 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:59.046960 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:59.047496 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:59.047499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:59.047500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:59.047583 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:59.047973 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:59.047976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:59.047977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:59.048172 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:59.048255 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:59.048784 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 06:24:59.048897 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260810 06:24:59.049754 5271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:59.036730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:46421 (local address 127.4.73.3:44017)
0810 06:24:59.037269 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 06:24:59.037315 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:24:59.038726 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:24:59.039300 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:24:59.039311 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:24:59.039886 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:24:59.040487 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:59.040503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.041430 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.041433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:59.042341 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:24:59.042351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.042827 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.042830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:24:59.043001 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:59.043569 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:24:59.043586 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:24:59.045119 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:24:59.047081 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:59.047086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:59.047098 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:59.047368 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:59.047692 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:24:59.047695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:24:59.047696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:24:59.047837 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:24:59.048264 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:24:59.048269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:24:59.048554 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 06:24:59.048703 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 06:24:59.048958 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":165,"threads_started":1}
I20260810 06:24:59.050909 5268 heartbeater.cc:347] Connected to a master server at 127.4.73.62:46421
I20260810 06:24:59.051138 5268 heartbeater.cc:464] Registering TS with master...
I20260810 06:24:59.051666 5268 heartbeater.cc:511] Master 127.4.73.62:46421 requested a full tablet report, sending...
I20260810 06:24:59.052738 4480 ts_manager.cc:194] Registered new tserver with Master: 9799a8e856a94dbfaf2f26810b948630 (127.4.73.3:38081)
I20260810 06:24:59.053309 4480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:44017
I20260810 06:24:59.064393 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:24:59.093338 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:59.083825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.4.73.1:42735)
0810 06:24:59.084185 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:59.084199 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:59.084222 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:24:59.084275 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:59.084280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:59.084542 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:59.084660 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:59.085319 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.086109 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.086943 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.087301 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.087394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:59.089796 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:59.089805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:59.089818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:59.089842 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:59.091514 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:59.091966 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:59.091971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:59.091973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:59.092052 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:59.092351 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:59.092355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:59.092357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:59.092595 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:59.092835 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:59.093007 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 06:24:59.093172 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
W20260810 06:24:59.094372 4657 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38474
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[4409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[4409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:24:59.121052 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:24:59.113259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.4.73.1:42735)
0810 06:24:59.113427 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 06:24:59.113430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:24:59.113483 (+ 53us) server_negotiation.cc:407] Connection header received
0810 06:24:59.113608 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:24:59.113611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:24:59.113653 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:24:59.113723 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:24:59.114416 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.114929 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.115740 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:24:59.115974 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:24:59.116054 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:24:59.118378 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:24:59.118381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:24:59.118383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:24:59.118404 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:24:59.119647 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:59.120029 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:59.120031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:59.120032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:59.120105 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:24:59.120368 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:24:59.120370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:24:59.120371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:24:59.120539 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:24:59.120642 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:24:59.120836 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 06:24:59.120933 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:24:59.122267 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 4585
I20260810 06:24:59.133526 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 4809
I20260810 06:24:59.158635 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5035
I20260810 06:24:59.174273 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 4432
2026-08-10T06:24:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2590 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[5295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[5295](info): set up 2 sockets
Aug 10 06:24:59 dist-test-slave-45dx krb5kdc[5295](info): commencing operation
krb5kdc: starting...
W20260810 06:25:00.461376 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:00 dist-test-slave-45dx krb5kdc[5295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:00.677353 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38011
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:38011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:00.798729 5314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:00.799053 5314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:00.799126 5314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:00.803097 5314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:00.803187 5314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:00.803224 5314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:00.803258 5314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:00.803290 5314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:00.808494 5314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:38011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:00.809545 5314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:00.810563 5314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:00.819150 5321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:00.819136 5322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:00.819268 5314 server_base.cc:1034] running on GCE node
W20260810 06:25:00.819136 5324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:00.820068 5314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:00.821151 5314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:00.822335 5314 hybrid_clock.cc:648] HybridClock initialized: now 1786343100822312 us; error 36 us; skew 500 ppm
Aug 10 06:25:00 dist-test-slave-45dx krb5kdc[5295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343100, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:00.828233 5314 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:00.829165 5314 webserver.cc:460] Webserver started at http://127.4.73.62:40351/ using document root <none> and password file <none>
I20260810 06:25:00.829756 5314 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:00.829859 5314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:00.830101 5314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:00.832235 5314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "61ca5d526e15408a8116fff5aa5df4a2"
format_stamp: "Formatted at 2026-08-10 06:25:00 on dist-test-slave-45dx"
server_key: "d83c06de31204608b22cf96890f4b9c0"
server_key_iv: "7b6e418119e1c5ed42569e2e8bef5297"
server_key_version: "encryptionkey@0"
I20260810 06:25:00.832887 5314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "61ca5d526e15408a8116fff5aa5df4a2"
format_stamp: "Formatted at 2026-08-10 06:25:00 on dist-test-slave-45dx"
server_key: "d83c06de31204608b22cf96890f4b9c0"
server_key_iv: "7b6e418119e1c5ed42569e2e8bef5297"
server_key_version: "encryptionkey@0"
I20260810 06:25:00.836781 5314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:25:00.839103 5338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:00.840296 5314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:00.840512 5314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "61ca5d526e15408a8116fff5aa5df4a2"
format_stamp: "Formatted at 2026-08-10 06:25:00 on dist-test-slave-45dx"
server_key: "d83c06de31204608b22cf96890f4b9c0"
server_key_iv: "7b6e418119e1c5ed42569e2e8bef5297"
server_key_version: "encryptionkey@0"
I20260810 06:25:00.840626 5314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:00.851598 5314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:00.854382 5314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:00.854580 5314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:00.861423 5314 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:38011
I20260810 06:25:00.861451 5436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:38011 every 8 connection(s)
I20260810 06:25:00.862404 5314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:00.865121 5437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:00.868626 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5314
I20260810 06:25:00.868728 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:00.868989 4388 external_mini_cluster.cc:1428] Setting key f2162cf41b0a6c229806d342bade93ea
I20260810 06:25:00.871093 5437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2: Bootstrap starting.
I20260810 06:25:00.874122 5437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:00.875079 5437 log.cc:824] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:00.877559 5437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2: No bootstrap required, opened a new log
Aug 10 06:25:00 dist-test-slave-45dx krb5kdc[5295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:00.881700 5437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ca5d526e15408a8116fff5aa5df4a2" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38011 } }
I20260810 06:25:00.882054 5437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:00.882119 5437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ca5d526e15408a8116fff5aa5df4a2, State: Initialized, Role: FOLLOWER
I20260810 06:25:00.882652 5437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ca5d526e15408a8116fff5aa5df4a2" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38011 } }
I20260810 06:25:00.882848 5437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:00.882925 5437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:00.883055 5437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:00.884354 5437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ca5d526e15408a8116fff5aa5df4a2" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38011 } }
I20260810 06:25:00.884737 5437 leader_election.cc:302] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61ca5d526e15408a8116fff5aa5df4a2; no voters:
I20260810 06:25:00.885119 5437 leader_election.cc:288] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:00.885236 5442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:00.885582 5442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [term 1 LEADER]: Becoming Leader. State: Replica: 61ca5d526e15408a8116fff5aa5df4a2, State: Running, Role: LEADER
I20260810 06:25:00.885951 5442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ca5d526e15408a8116fff5aa5df4a2" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38011 } }
I20260810 06:25:00.886360 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:00.870370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.4.73.62:38011)
0810 06:25:00.870787 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:00.870810 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:00.870852 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:25:00.871685 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:00.871709 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:00.872114 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:00.872547 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:00.873321 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:00.874891 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:00.876007 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:00.876749 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:00.876931 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:00.879620 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:00.879627 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:00.879646 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:00.879683 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:00.882864 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:00.883408 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:00.883414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:00.883420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:00.883571 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:00.883857 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:00.883860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:00.883862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:00.884526 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:00.884723 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:00.885055 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 06:25:00.885387 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":101,"threads_started":1}
I20260810 06:25:00.886523 5437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:00.888664 5445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61ca5d526e15408a8116fff5aa5df4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ca5d526e15408a8116fff5aa5df4a2" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38011 } } }
I20260810 06:25:00.888818 5445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:00.889276 5469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:00.889240 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61ca5d526e15408a8116fff5aa5df4a2. Latest consensus state: current_term: 1 leader_uuid: "61ca5d526e15408a8116fff5aa5df4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61ca5d526e15408a8116fff5aa5df4a2" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38011 } } }
I20260810 06:25:00.889590 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:00.892206 5469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:00.898003 5469 catalog_manager.cc:1347] Generated new cluster ID: 166b8b32f9684db29d143509f96a480d
I20260810 06:25:00.898077 5469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:00.921495 5469 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:00.922446 5469 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:00.929406 5469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61ca5d526e15408a8116fff5aa5df4a2: Generated new TSK 0
I20260810 06:25:00.930219 5469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:00.969111 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38011
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:01.084275 5477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:01.084566 5477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:01.084657 5477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:01.088912 5477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:01.088982 5477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:01.089044 5477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:01.094400 5477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:01.095559 5477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:01.096673 5477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:01.106917 5486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:01.106917 5484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:01.106928 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:01.107923 5477 server_base.cc:1034] running on GCE node
I20260810 06:25:01.108364 5477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:01.109000 5477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:01.110177 5477 hybrid_clock.cc:648] HybridClock initialized: now 1786343101110140 us; error 50 us; skew 500 ppm
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:01.113195 5477 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:01.114122 5477 webserver.cc:460] Webserver started at http://127.4.73.1:36079/ using document root <none> and password file <none>
I20260810 06:25:01.114701 5477 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:01.114792 5477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:01.115031 5477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:01.117233 5477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "8357615993c74a3c9424daaae8df4833"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "c4a96c133f7d25d529c8abbf92f263dc"
server_key_iv: "48f39df66691f90a35f36e5085703b21"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.117820 5477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "8357615993c74a3c9424daaae8df4833"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "c4a96c133f7d25d529c8abbf92f263dc"
server_key_iv: "48f39df66691f90a35f36e5085703b21"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.121587 5477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:25:01.123905 5495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.125087 5477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:01.125221 5477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "8357615993c74a3c9424daaae8df4833"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "c4a96c133f7d25d529c8abbf92f263dc"
server_key_iv: "48f39df66691f90a35f36e5085703b21"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.125327 5477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:01.139135 5477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:01.142321 5477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:01.142513 5477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:01.143070 5477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:01.143929 5477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:01.143997 5477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.144068 5477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:01.144107 5477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.153385 5477 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:35301
I20260810 06:25:01.153421 5698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:35301 every 8 connection(s)
I20260810 06:25:01.154274 5477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:01.156032 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5477
I20260810 06:25:01.156148 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:01.156471 4388 external_mini_cluster.cc:1428] Setting key ee83463915570fff03e28195b8d849f6
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:25:01.168609 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.156129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:42615 (local address 127.4.73.62:38011)
0810 06:25:01.156277 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.156281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.157285 (+ 1004us) server_negotiation.cc:407] Connection header received
0810 06:25:01.158707 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.158713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.158771 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.158862 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:01.159972 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.160556 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.161718 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.161961 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.162073 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.164977 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:01.164980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:01.164982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:01.165006 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:01.166542 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.167039 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.167043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.167044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.167128 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.167496 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.167499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.167501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.167692 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:01.167799 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.168268 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.168433 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:01.169539 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.156605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38011 (local address 127.4.73.1:42615)
0810 06:25:01.157115 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.157167 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.158482 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.159013 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.159019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.159348 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:01.159821 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.159835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.160677 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.160680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.161586 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.161595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.162113 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.162117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.162294 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.162925 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:01.162942 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:01.164805 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:01.166661 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:01.166666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:01.166679 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:01.166939 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:01.167246 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:01.167250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:01.167252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:01.167400 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:01.167798 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:01.167805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:01.168040 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.168224 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.168567 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":182,"threads_started":1}
I20260810 06:25:01.170768 5699 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38011
I20260810 06:25:01.171029 5699 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:01.171574 5699 heartbeater.cc:511] Master 127.4.73.62:38011 requested a full tablet report, sending...
I20260810 06:25:01.173077 5370 ts_manager.cc:194] Registered new tserver with Master: 8357615993c74a3c9424daaae8df4833 (127.4.73.1:35301)
I20260810 06:25:01.174355 5370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:42615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:01.184804 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38011
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:01.298326 5705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:01.298635 5705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:01.298681 5705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:01.302861 5705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:01.302929 5705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:01.303007 5705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:01.308210 5705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:01.309211 5705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:01.310163 5705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:01.319993 5716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:01.320055 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:01.320063 5705 server_base.cc:1034] running on GCE node
W20260810 06:25:01.320017 5718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:01.320771 5705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:01.321388 5705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:01.322572 5705 hybrid_clock.cc:648] HybridClock initialized: now 1786343101322541 us; error 45 us; skew 500 ppm
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:01.325341 5705 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:01.326229 5705 webserver.cc:460] Webserver started at http://127.4.73.2:34275/ using document root <none> and password file <none>
I20260810 06:25:01.326793 5705 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:01.326886 5705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:01.327131 5705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:01.329386 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "cdb3c5a99340487a8689fc250c9a8657"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "07a098f0806c64495d909eddb9deb201"
server_key_iv: "ad5e5afc94d32efd529e8ca2fe6aee9b"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.329994 5705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "cdb3c5a99340487a8689fc250c9a8657"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "07a098f0806c64495d909eddb9deb201"
server_key_iv: "ad5e5afc94d32efd529e8ca2fe6aee9b"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.333843 5705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:25:01.336530 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.337807 5705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:01.337944 5705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "cdb3c5a99340487a8689fc250c9a8657"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "07a098f0806c64495d909eddb9deb201"
server_key_iv: "ad5e5afc94d32efd529e8ca2fe6aee9b"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.338066 5705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:01.352205 5705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:01.355122 5705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:01.355324 5705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:01.355916 5705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:01.356868 5705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:01.356933 5705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.356998 5705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:01.357040 5705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.366320 5705 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:46393
I20260810 06:25:01.366353 5908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:46393 every 8 connection(s)
I20260810 06:25:01.367208 5705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:01.371613 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5705
I20260810 06:25:01.371706 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:01.371966 4388 external_mini_cluster.cc:1428] Setting key 2d8ab2daaa464e6377bab4f793f4982b
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:01.381935 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.368989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:50525 (local address 127.4.73.62:38011)
0810 06:25:01.369135 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.369139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.369983 (+ 844us) server_negotiation.cc:407] Connection header received
0810 06:25:01.371360 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.371363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.371422 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.371510 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:01.372942 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.373468 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.374801 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.375033 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.375140 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.377723 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:01.377730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:01.377736 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:01.377770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:01.379618 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.380139 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.380145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.380148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.380254 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.380637 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.380641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.380642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.380918 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:01.381029 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.381565 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.381706 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:25:01.382655 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.369368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38011 (local address 127.4.73.2:50525)
0810 06:25:01.369827 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.369876 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.371141 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.371703 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.371713 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.372148 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:01.372777 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.372794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.373594 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.373598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.374672 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.374682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.375181 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.375185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.375366 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.375976 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:01.375993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:01.377557 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:01.379740 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:01.379745 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:01.379756 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:01.380035 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:01.380388 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:01.380391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:01.380392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:01.380538 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:01.381028 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:01.381033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:01.381307 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.381473 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.381738 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:25:01.383821 5909 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38011
I20260810 06:25:01.384092 5909 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:01.384690 5909 heartbeater.cc:511] Master 127.4.73.62:38011 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:25:01.385906 5370 ts_manager.cc:194] Registered new tserver with Master: cdb3c5a99340487a8689fc250c9a8657 (127.4.73.2:46393)
I20260810 06:25:01.386705 5370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:50525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:01.403051 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38011
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:01.517213 5919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:01.517447 5919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:01.517489 5919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:01.521620 5919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:01.521682 5919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:01.521800 5919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:01.527060 5919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38011
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:01.528162 5919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:01.529237 5919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:01.539558 5924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:01.539601 5928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:01.539624 5919 server_base.cc:1034] running on GCE node
W20260810 06:25:01.539561 5926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:01.540241 5919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:01.540904 5919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:01.542070 5919 hybrid_clock.cc:648] HybridClock initialized: now 1786343101542054 us; error 29 us; skew 500 ppm
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:01.545059 5919 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:01.545902 5919 webserver.cc:460] Webserver started at http://127.4.73.3:37995/ using document root <none> and password file <none>
I20260810 06:25:01.546454 5919 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:01.546545 5919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:01.546773 5919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:01.548841 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "94843ddafb8e46ad89e1e23b7c36bdef"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "1637d0c29c342b36591cb13672cd1280"
server_key_iv: "22de91fa9b9a0f265e6abfe8ddd0f659"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.549434 5919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "94843ddafb8e46ad89e1e23b7c36bdef"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "1637d0c29c342b36591cb13672cd1280"
server_key_iv: "22de91fa9b9a0f265e6abfe8ddd0f659"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.553121 5919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:25:01.555462 5942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.556713 5919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:01.556841 5919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "94843ddafb8e46ad89e1e23b7c36bdef"
format_stamp: "Formatted at 2026-08-10 06:25:01 on dist-test-slave-45dx"
server_key: "1637d0c29c342b36591cb13672cd1280"
server_key_iv: "22de91fa9b9a0f265e6abfe8ddd0f659"
server_key_version: "encryptionkey@0"
I20260810 06:25:01.556967 5919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:01.573880 5919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:01.576833 5919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:01.577041 5919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:01.577612 5919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:01.578421 5919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:01.578492 5919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.578578 5919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:01.578624 5919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:01.588369 5919 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:42449
I20260810 06:25:01.588383 6116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:42449 every 8 connection(s)
I20260810 06:25:01.589268 5919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:01.589745 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5919
I20260810 06:25:01.589898 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:01.590230 4388 external_mini_cluster.cc:1428] Setting key 3c1dfae8b61e011c73369b1c58e738aa
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:01.602942 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.591157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:43075 (local address 127.4.73.62:38011)
0810 06:25:01.591310 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.591316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.592290 (+ 974us) server_negotiation.cc:407] Connection header received
0810 06:25:01.593762 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.593765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.593822 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.593921 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:01.595060 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.595592 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.596826 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.597057 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.597167 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.599514 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:01.599518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:01.599525 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:01.599549 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:01.600878 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.601380 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.601384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.601386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.601465 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.601802 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.601806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.601808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.602043 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:01.602154 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.602633 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.602762 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:25:01.603892 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.591661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38011 (local address 127.4.73.3:43075)
0810 06:25:01.592157 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.592198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.593551 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.594095 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.594101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.594455 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:01.594919 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.594932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.595715 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.595719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.596703 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.596713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.597204 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.597209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.597384 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.597917 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:01.597933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:01.599339 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:01.600995 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:01.601001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:01.601014 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:01.601268 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:01.601557 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:01.601561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:01.601563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:01.601703 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:01.602154 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:01.602161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:01.602407 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.602591 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.602905 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260810 06:25:01.605027 6118 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38011
I20260810 06:25:01.605228 6118 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:01.605683 6118 heartbeater.cc:511] Master 127.4.73.62:38011 requested a full tablet report, sending...
I20260810 06:25:01.606590 5370 ts_manager.cc:194] Registered new tserver with Master: 94843ddafb8e46ad89e1e23b7c36bdef (127.4.73.3:42449)
I20260810 06:25:01.607110 5370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:43075
I20260810 06:25:01.616556 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:25:01.630555 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.621451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.4.73.62:38011)
0810 06:25:01.621616 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.621624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.621762 (+ 138us) server_negotiation.cc:407] Connection header received
0810 06:25:01.621887 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.621890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.621965 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.622069 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:01.622753 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.623484 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.624348 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.624643 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.624731 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.626666 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:01.626669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:01.626670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:01.626693 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:01.628204 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.628825 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.628832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.628837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.628959 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.629292 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.629296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.629298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.629702 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:01.629860 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.630162 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.630308 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:25:01.636831 5370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:01.640580 5370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:01.658910 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.652109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35301 (local address 127.0.0.1:41792)
0810 06:25:01.652926 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.652977 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.653154 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.653954 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.653960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.653985 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:01.654113 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.654119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.655599 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.655602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.657013 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.657024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.658356 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.658358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.658435 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.658463 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.658620 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.658772 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260810 06:25:01.659447 6134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.652359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:46393 (local address 127.0.0.1:41070)
0810 06:25:01.652946 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.653005 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.653137 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.654392 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.654395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.654413 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:01.654523 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.654534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.656058 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.656062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.657601 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.657616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.658980 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.658985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.659081 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.659105 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.659184 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.659262 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":39,"thread_start_us":125,"threads_started":1}
I20260810 06:25:01.659605 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.653203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:42449 (local address 127.0.0.1:35640)
0810 06:25:01.653503 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.653519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.653610 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.654543 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.654546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.654565 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:01.654651 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.654657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.656971 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.656974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.658124 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.658134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.659245 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.659248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.659334 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.659356 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.659403 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.659468 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":165,"threads_started":1}
I20260810 06:25:01.660153 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.652768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.4.73.2:46393)
0810 06:25:01.653932 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.653943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.653965 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:25:01.654034 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.654038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.654258 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.654409 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:01.654635 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.655876 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.657764 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.658667 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.658771 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.658905 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.659900 (+ 995us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.659995 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":208,"threads_started":1}
I20260810 06:25:01.660149 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.652274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.4.73.1:35301)
0810 06:25:01.652679 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.652696 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.653070 (+ 374us) server_negotiation.cc:407] Connection header received
0810 06:25:01.653356 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.653363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.653582 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.653739 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:01.654241 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.655442 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.657155 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.657979 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.658084 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.658299 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.659300 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.659410 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":173,"threads_started":1}
I20260810 06:25:01.663496 5589 tablet_service.cc:1467] Processing CreateTablet for tablet c1307575c84041d5bc56636673f9e2a8 (DEFAULT_TABLE table=test-table [id=05740f9203fe4dce9adb173827c85222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:01.663429 5804 tablet_service.cc:1467] Processing CreateTablet for tablet c1307575c84041d5bc56636673f9e2a8 (DEFAULT_TABLE table=test-table [id=05740f9203fe4dce9adb173827c85222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:01.664731 5804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1307575c84041d5bc56636673f9e2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:01.664839 5589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1307575c84041d5bc56636673f9e2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:01.668438 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.653713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.4.73.3:42449)
0810 06:25:01.654048 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.654065 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.654088 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:25:01.654163 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.654166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.654382 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.654795 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:01.654953 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.656819 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.658254 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.659119 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.659882 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.660018 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.660065 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.660144 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":118,"threads_started":1}
I20260810 06:25:01.671667 6010 tablet_service.cc:1467] Processing CreateTablet for tablet c1307575c84041d5bc56636673f9e2a8 (DEFAULT_TABLE table=test-table [id=05740f9203fe4dce9adb173827c85222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:01.672930 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1307575c84041d5bc56636673f9e2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:01.673226 6141 tablet_bootstrap.cc:492] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833: Bootstrap starting.
I20260810 06:25:01.674510 6142 tablet_bootstrap.cc:492] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657: Bootstrap starting.
I20260810 06:25:01.676035 6141 tablet_bootstrap.cc:654] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:01.677069 6141 log.cc:824] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:01.678056 6142 tablet_bootstrap.cc:654] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:01.679304 6142 log.cc:824] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:01.681684 6147 tablet_bootstrap.cc:492] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef: Bootstrap starting.
I20260810 06:25:01.681869 6142 tablet_bootstrap.cc:492] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657: No bootstrap required, opened a new log
I20260810 06:25:01.681864 6141 tablet_bootstrap.cc:492] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833: No bootstrap required, opened a new log
I20260810 06:25:01.682056 6141 ts_tablet_manager.cc:1397] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 06:25:01.682056 6142 ts_tablet_manager.cc:1397] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 06:25:01.684055 6147 tablet_bootstrap.cc:654] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:01.685559 6147 log.cc:824] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:01.688241 6147 tablet_bootstrap.cc:492] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef: No bootstrap required, opened a new log
I20260810 06:25:01.688625 6147 ts_tablet_manager.cc:1397] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:25:01.688235 6142 raft_consensus.cc:348] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.688730 6142 raft_consensus.cc:374] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:01.688794 6142 raft_consensus.cc:729] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb3c5a99340487a8689fc250c9a8657, State: Initialized, Role: FOLLOWER
I20260810 06:25:01.689397 6142 consensus_queue.cc:260] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.690446 5909 heartbeater.cc:503] Master 127.4.73.62:38011 was elected leader, sending a full tablet report...
I20260810 06:25:01.690693 6142 ts_tablet_manager.cc:1428] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260810 06:25:01.690908 6141 raft_consensus.cc:348] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.691303 6141 raft_consensus.cc:374] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:01.691425 6141 raft_consensus.cc:729] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8357615993c74a3c9424daaae8df4833, State: Initialized, Role: FOLLOWER
I20260810 06:25:01.692052 6141 consensus_queue.cc:260] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.692984 5699 heartbeater.cc:503] Master 127.4.73.62:38011 was elected leader, sending a full tablet report...
I20260810 06:25:01.693285 6141 ts_tablet_manager.cc:1428] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260810 06:25:01.696779 6147 raft_consensus.cc:348] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.697192 6147 raft_consensus.cc:374] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:01.697260 6147 raft_consensus.cc:729] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94843ddafb8e46ad89e1e23b7c36bdef, State: Initialized, Role: FOLLOWER
I20260810 06:25:01.697641 6147 consensus_queue.cc:260] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.698391 6118 heartbeater.cc:503] Master 127.4.73.62:38011 was elected leader, sending a full tablet report...
I20260810 06:25:01.698566 6147 ts_tablet_manager.cc:1428] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
W20260810 06:25:01.840512 6120 tablet.cc:2369] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:01.843312 6155 raft_consensus.cc:482] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:01.843556 6155 raft_consensus.cc:504] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.844991 6155 leader_election.cc:288] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8357615993c74a3c9424daaae8df4833 (127.4.73.1:35301), cdb3c5a99340487a8689fc250c9a8657 (127.4.73.2:46393)
I20260810 06:25:01.849112 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.845414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:35245 (local address 127.4.73.2:46393)
0810 06:25:01.845560 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.845565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.845582 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:25:01.845676 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.845679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.845733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.845809 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:01.846213 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.846915 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.848071 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.848549 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.848668 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.848737 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.848909 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.848971 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:25:01.849126 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.845318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:46393 (local address 127.4.73.3:35245)
0810 06:25:01.845444 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.845461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.845572 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.845930 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.845934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.845959 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:01.846105 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.846114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.847024 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.847029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.847947 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.847957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.848670 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.848674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.848788 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.848807 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.848894 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.848970 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 06:25:01.849165 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.845420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:52407 (local address 127.4.73.1:35301)
0810 06:25:01.845547 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.845552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.846067 (+ 515us) server_negotiation.cc:407] Connection header received
0810 06:25:01.846163 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.846166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.846219 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.846313 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:01.846598 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.847102 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.848163 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.848610 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.848720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.848788 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.848970 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.849041 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:25:01.849180 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.845318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35301 (local address 127.4.73.3:52407)
0810 06:25:01.845944 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:01.845957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:01.846072 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:01.846324 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:01.846327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:01.846349 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:01.846470 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.846477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.847214 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.847219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.848040 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:01.848050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.848727 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.848730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:01.848852 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.848870 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:25:01.848955 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 06:25:01.849024 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":257,"thread_start_us":147,"threads_started":1}
I20260810 06:25:01.849920 5840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1307575c84041d5bc56636673f9e2a8" candidate_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb3c5a99340487a8689fc250c9a8657" is_pre_election: true
I20260810 06:25:01.849970 5619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1307575c84041d5bc56636673f9e2a8" candidate_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8357615993c74a3c9424daaae8df4833" is_pre_election: true
I20260810 06:25:01.850291 5840 raft_consensus.cc:2454] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94843ddafb8e46ad89e1e23b7c36bdef in term 0.
I20260810 06:25:01.850297 5619 raft_consensus.cc:2454] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94843ddafb8e46ad89e1e23b7c36bdef in term 0.
I20260810 06:25:01.850765 5943 leader_election.cc:302] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94843ddafb8e46ad89e1e23b7c36bdef, cdb3c5a99340487a8689fc250c9a8657; no voters:
I20260810 06:25:01.850992 6155 raft_consensus.cc:2781] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:01.851100 6155 raft_consensus.cc:482] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:01.851161 6155 raft_consensus.cc:3037] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:01.852226 6155 raft_consensus.cc:504] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.852674 6155 leader_election.cc:288] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [CANDIDATE]: Term 1 election: Requested vote from peers 8357615993c74a3c9424daaae8df4833 (127.4.73.1:35301), cdb3c5a99340487a8689fc250c9a8657 (127.4.73.2:46393)
I20260810 06:25:01.853063 5840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1307575c84041d5bc56636673f9e2a8" candidate_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb3c5a99340487a8689fc250c9a8657"
I20260810 06:25:01.853082 5619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1307575c84041d5bc56636673f9e2a8" candidate_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8357615993c74a3c9424daaae8df4833"
I20260810 06:25:01.853201 5619 raft_consensus.cc:3037] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:01.853199 5840 raft_consensus.cc:3037] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:01.854524 5840 raft_consensus.cc:2454] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94843ddafb8e46ad89e1e23b7c36bdef in term 1.
I20260810 06:25:01.854530 5619 raft_consensus.cc:2454] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94843ddafb8e46ad89e1e23b7c36bdef in term 1.
I20260810 06:25:01.854830 5943 leader_election.cc:302] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94843ddafb8e46ad89e1e23b7c36bdef, cdb3c5a99340487a8689fc250c9a8657; no voters:
I20260810 06:25:01.855046 6155 raft_consensus.cc:2781] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:01.855288 6155 raft_consensus.cc:686] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [term 1 LEADER]: Becoming Leader. State: Replica: 94843ddafb8e46ad89e1e23b7c36bdef, State: Running, Role: LEADER
I20260810 06:25:01.855640 6155 consensus_queue.cc:237] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } }
I20260810 06:25:01.859298 5368 catalog_manager.cc:5495] T c1307575c84041d5bc56636673f9e2a8 P 94843ddafb8e46ad89e1e23b7c36bdef reported cstate change: term changed from 0 to 1, leader changed from <none> to 94843ddafb8e46ad89e1e23b7c36bdef (127.4.73.3). New cstate: current_term: 1 leader_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8357615993c74a3c9424daaae8df4833" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdb3c5a99340487a8689fc250c9a8657" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94843ddafb8e46ad89e1e23b7c36bdef" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42449 } health_report { overall_health: HEALTHY } } }
W20260810 06:25:01.868608 5912 tablet.cc:2369] T c1307575c84041d5bc56636673f9e2a8 P cdb3c5a99340487a8689fc250c9a8657: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:01.904505 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.894294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.4.73.1:35301)
0810 06:25:01.894449 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.894453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.894468 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:25:01.894577 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.894581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.894639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.894730 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:01.895428 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.895951 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.896868 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.897110 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.897219 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.900032 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:01.900046 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:01.900063 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:01.900090 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:01.902759 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.903177 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.903183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.903184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.903271 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.903464 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.903467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.903468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.903725 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:01.904041 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.904139 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.904252 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 06:25:01.905580 5700 tablet.cc:2369] T c1307575c84041d5bc56636673f9e2a8 P 8357615993c74a3c9424daaae8df4833: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:01.938189 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.928179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.4.73.1:35301)
0810 06:25:01.928422 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.928428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.928636 (+ 208us) server_negotiation.cc:407] Connection header received
0810 06:25:01.928881 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.928887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.928951 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.929054 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:01.929911 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.930452 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.931284 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.931527 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.931640 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.934573 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:01.934576 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:01.934578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:01.934606 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:01.936003 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.936580 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.936585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.936587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.936724 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.937113 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.937119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.937123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.937433 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:01.937585 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.937896 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.938054 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:25:01 dist-test-slave-45dx krb5kdc[5295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:01.971323 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:01.960745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.4.73.1:35301)
0810 06:25:01.960937 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:01.960941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:01.961081 (+ 140us) server_negotiation.cc:407] Connection header received
0810 06:25:01.961334 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:01.961338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:01.961401 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:01.961538 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:01.962593 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.963140 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.964392 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:01.964624 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:01.964707 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:01.967681 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:01.967684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:01.967686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:01.967713 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:01.969416 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.969985 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.969991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.969994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.970100 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:01.970421 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:01.970426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:01.970430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:01.970615 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:01.970721 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:01.971068 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 06:25:01.971176 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 06:25:01.973376 5551 tablet_service.cc:2908] Rejecting scan request for tablet c1307575c84041d5bc56636673f9e2a8: Uninitialized: clean time has not yet been initialized
I20260810 06:25:01.976372 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5477
I20260810 06:25:02.000864 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5705
I20260810 06:25:02.012398 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5919
I20260810 06:25:02.023521 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5314
2026-08-10T06:25:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2860 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:02 dist-test-slave-45dx krb5kdc[6181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:02 dist-test-slave-45dx krb5kdc[6181](info): set up 2 sockets
Aug 10 06:25:02 dist-test-slave-45dx krb5kdc[6181](info): commencing operation
krb5kdc: starting...
W20260810 06:25:03.355841 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:03 dist-test-slave-45dx krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:03.487545 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44241
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:44241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:03.624119 6202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:03.624672 6202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:03.624778 6202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:03.629637 6202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:03.629721 6202 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 06:25:03.629751 6202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:03.629783 6202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:03.629804 6202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:03.629820 6202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:03.636193 6202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:44241
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:03.637331 6202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:03.638531 6202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:03.648808 6210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:03.648860 6214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:03.648861 6211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:03.649058 6202 server_base.cc:1034] running on GCE node
I20260810 06:25:03.649699 6202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:03.650786 6202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:03.651970 6202 hybrid_clock.cc:648] HybridClock initialized: now 1786343103651937 us; error 43 us; skew 500 ppm
Aug 10 06:25:03 dist-test-slave-45dx krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:03.654932 6202 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:03.655792 6202 webserver.cc:460] Webserver started at http://127.4.73.62:40503/ using document root <none> and password file <none>
I20260810 06:25:03.656391 6202 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:03.656458 6202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:03.656687 6202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:03.659049 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "93f1544b811946ab8610df1bebf2e1f5"
format_stamp: "Formatted at 2026-08-10 06:25:03 on dist-test-slave-45dx"
server_key: "728d0deaf528d2210f68968776dfe365"
server_key_iv: "42789e8b69f2e528a17b9731a303aec3"
server_key_version: "encryptionkey@0"
I20260810 06:25:03.659639 6202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "93f1544b811946ab8610df1bebf2e1f5"
format_stamp: "Formatted at 2026-08-10 06:25:03 on dist-test-slave-45dx"
server_key: "728d0deaf528d2210f68968776dfe365"
server_key_iv: "42789e8b69f2e528a17b9731a303aec3"
server_key_version: "encryptionkey@0"
I20260810 06:25:03.663735 6202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 06:25:03.666404 6227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:03.667728 6202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:03.667932 6202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "93f1544b811946ab8610df1bebf2e1f5"
format_stamp: "Formatted at 2026-08-10 06:25:03 on dist-test-slave-45dx"
server_key: "728d0deaf528d2210f68968776dfe365"
server_key_iv: "42789e8b69f2e528a17b9731a303aec3"
server_key_version: "encryptionkey@0"
I20260810 06:25:03.668081 6202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:03.686642 6202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:03.689980 6202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:03.690200 6202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:03.701157 6202 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:44241
I20260810 06:25:03.701161 6342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:44241 every 8 connection(s)
I20260810 06:25:03.702311 6202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:03.704429 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6202
I20260810 06:25:03.704546 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:03.704850 4388 external_mini_cluster.cc:1428] Setting key 58a727c0df02f80b2542bcad5cf5c94f
I20260810 06:25:03.706650 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:03.713259 6343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5: Bootstrap starting.
Aug 10 06:25:03 dist-test-slave-45dx krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:03.715870 6343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:03.717321 6343 log.cc:824] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:03.719185 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:03.706185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.4.73.62:44241)
0810 06:25:03.706621 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:03.706639 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:03.706672 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:25:03.707358 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:03.707374 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:03.707689 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:03.708033 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:03.709036 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:03.710045 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:03.710926 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:03.711398 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:03.711575 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:03.714222 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:03.714227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:03.714239 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:03.714266 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:03.716025 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:03.716533 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:03.716538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:03.716542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:03.716659 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:03.716965 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:03.716968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:03.716969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:03.717361 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:03.717641 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:03.717940 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 06:25:03.718265 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":135,"threads_started":1}
I20260810 06:25:03.719637 6343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5: No bootstrap required, opened a new log
I20260810 06:25:03.723083 6343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f1544b811946ab8610df1bebf2e1f5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44241 } }
I20260810 06:25:03.723304 6343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:03.723340 6343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f1544b811946ab8610df1bebf2e1f5, State: Initialized, Role: FOLLOWER
I20260810 06:25:03.723727 6343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f1544b811946ab8610df1bebf2e1f5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44241 } }
I20260810 06:25:03.723855 6343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:03.723901 6343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:03.723969 6343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:03.724890 6343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f1544b811946ab8610df1bebf2e1f5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44241 } }
I20260810 06:25:03.725174 6343 leader_election.cc:302] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f1544b811946ab8610df1bebf2e1f5; no voters:
I20260810 06:25:03.725430 6343 leader_election.cc:288] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:03.725561 6351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:03.725778 6351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [term 1 LEADER]: Becoming Leader. State: Replica: 93f1544b811946ab8610df1bebf2e1f5, State: Running, Role: LEADER
I20260810 06:25:03.726099 6351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f1544b811946ab8610df1bebf2e1f5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44241 } }
I20260810 06:25:03.726883 6343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:03.727965 6352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f1544b811946ab8610df1bebf2e1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f1544b811946ab8610df1bebf2e1f5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44241 } } }
I20260810 06:25:03.728107 6352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:03.728392 6353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f1544b811946ab8610df1bebf2e1f5. Latest consensus state: current_term: 1 leader_uuid: "93f1544b811946ab8610df1bebf2e1f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f1544b811946ab8610df1bebf2e1f5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44241 } } }
I20260810 06:25:03.728492 6353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:03.728593 6373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:03.731590 6373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:03.737885 6373 catalog_manager.cc:1347] Generated new cluster ID: 70ba5c56c0d940088defcc51e89c2763
I20260810 06:25:03.737949 6373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:03.775180 6373 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:03.776566 6373 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:03.782833 6373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5: Generated new TSK 0
I20260810 06:25:03.783696 6373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:03.819486 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44241
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:03.951759 6387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:03.952023 6387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:03.952131 6387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:03.957294 6387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:03.957357 6387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:03.957396 6387 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:03.957429 6387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:03.963295 6387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:03.964632 6387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:03.965786 6387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:03.977363 6398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:03.977375 6395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:03.977384 6396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:03.977767 6387 server_base.cc:1034] running on GCE node
I20260810 06:25:03.978267 6387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:03.979072 6387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:03.980281 6387 hybrid_clock.cc:648] HybridClock initialized: now 1786343103980262 us; error 36 us; skew 500 ppm
Aug 10 06:25:03 dist-test-slave-45dx krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:03.983225 6387 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:03.984349 6387 webserver.cc:460] Webserver started at http://127.4.73.1:35967/ using document root <none> and password file <none>
I20260810 06:25:03.985028 6387 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:03.985183 6387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:03.985497 6387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:03.988245 6387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "b7f686ba1dda43fb800e3d199d89cb4b"
format_stamp: "Formatted at 2026-08-10 06:25:03 on dist-test-slave-45dx"
server_key: "94df7041997b11fd7b5d926fa920a52f"
server_key_iv: "13e14011e94cf722e924b6368448a565"
server_key_version: "encryptionkey@0"
I20260810 06:25:03.988965 6387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "b7f686ba1dda43fb800e3d199d89cb4b"
format_stamp: "Formatted at 2026-08-10 06:25:03 on dist-test-slave-45dx"
server_key: "94df7041997b11fd7b5d926fa920a52f"
server_key_iv: "13e14011e94cf722e924b6368448a565"
server_key_version: "encryptionkey@0"
I20260810 06:25:03.993857 6387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 06:25:03.996378 6406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:03.998131 6387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:25:03.998257 6387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "b7f686ba1dda43fb800e3d199d89cb4b"
format_stamp: "Formatted at 2026-08-10 06:25:03 on dist-test-slave-45dx"
server_key: "94df7041997b11fd7b5d926fa920a52f"
server_key_iv: "13e14011e94cf722e924b6368448a565"
server_key_version: "encryptionkey@0"
I20260810 06:25:03.998364 6387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:04.016444 6387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:04.019761 6387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:04.019968 6387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:04.021646 6387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:04.021697 6387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:04.021773 6387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:04.021844 6387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:04 dist-test-slave-45dx krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:04.035787 6387 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:40319
I20260810 06:25:04.035976 6613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:40319 every 8 connection(s)
I20260810 06:25:04.036832 6387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:04.037299 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6387
I20260810 06:25:04.037446 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:04.037838 4388 external_mini_cluster.cc:1428] Setting key bef55a6bb3513bd75177b845830a8f05
I20260810 06:25:04.039463 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.022610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:57155 (local address 127.4.73.62:44241)
0810 06:25:04.022855 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.022862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.023864 (+ 1002us) server_negotiation.cc:407] Connection header received
0810 06:25:04.025201 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.025205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.025270 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.025372 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:04.026946 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.027878 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.029697 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.030061 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.030181 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.034655 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:04.034662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:04.034666 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:04.034701 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:04.036951 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.037600 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.037603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.037605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.037703 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.038219 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.038222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.038223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.038485 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:04.038620 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.039151 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.039286 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 06:25:04.040568 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.023141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44241 (local address 127.4.73.1:57155)
0810 06:25:04.023703 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.023750 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.024953 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.025555 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.025562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.026055 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:04.026786 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.026806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.028007 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.028013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.029551 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.029563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.030263 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.030268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.030495 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.031337 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:04.031356 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:04.034469 (+ 3113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:04.037098 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.037103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.037113 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.037475 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.037840 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.037859 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.037862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.038104 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.038612 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:04.038617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:04.038879 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.039061 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.039438 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":143,"threads_started":1}
I20260810 06:25:04.041962 6614 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44241
I20260810 06:25:04.042124 6614 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:04.042758 6614 heartbeater.cc:511] Master 127.4.73.62:44241 requested a full tablet report, sending...
I20260810 06:25:04.045310 6259 ts_manager.cc:194] Registered new tserver with Master: b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319)
I20260810 06:25:04.046957 6259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:57155
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:04.055317 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.044678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:36171 (local address 127.4.73.62:44241)
0810 06:25:04.044795 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.044799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.045249 (+ 450us) server_negotiation.cc:407] Connection header received
0810 06:25:04.045368 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.045371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.045507 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.045598 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:04.046198 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.046947 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.049990 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.050258 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.050397 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.051567 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:04.051570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:04.051573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:04.051596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:04.053185 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.053708 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.053711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.053713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.053795 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.054134 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.054139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.054141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.054501 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:04.054612 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.055053 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.055174 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:25:04.055552 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.044819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44241 (local address 127.4.73.1:36171)
0810 06:25:04.045168 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.045184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.045287 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.045643 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.045649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.045916 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:04.046065 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.046076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.047067 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.047072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.049864 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.049875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.050401 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.050405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.050520 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.051009 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:04.051027 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:04.051425 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:04.053328 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.053331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.053334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.053584 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.053939 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.053942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.053944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.054041 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.054628 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:04.054635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:04.054781 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.054952 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.055175 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:04.070645 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44241
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:04.195480 6631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:04.195779 6631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:04.195863 6631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:04.200368 6631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:04.200450 6631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:04.200515 6631 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:04.200587 6631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:04.206432 6631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:04.207535 6631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:04.208813 6631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:04.219551 6643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:04.219514 6647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:04.219575 6641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:04.219974 6631 server_base.cc:1034] running on GCE node
I20260810 06:25:04.220464 6631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:04.221298 6631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:04.222494 6631 hybrid_clock.cc:648] HybridClock initialized: now 1786343104222446 us; error 43 us; skew 500 ppm
Aug 10 06:25:04 dist-test-slave-45dx krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:04.225798 6631 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:04.226840 6631 webserver.cc:460] Webserver started at http://127.4.73.2:36277/ using document root <none> and password file <none>
I20260810 06:25:04.227598 6631 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:04.227686 6631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:04.227865 6631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:04.230618 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "82714ede57e64b70a9a7acb4c2fea98c"
format_stamp: "Formatted at 2026-08-10 06:25:04 on dist-test-slave-45dx"
server_key: "2c704ac520807514d4c2125edb8f5055"
server_key_iv: "e8957a5860482e66aecc8a19326ba13d"
server_key_version: "encryptionkey@0"
I20260810 06:25:04.231272 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "82714ede57e64b70a9a7acb4c2fea98c"
format_stamp: "Formatted at 2026-08-10 06:25:04 on dist-test-slave-45dx"
server_key: "2c704ac520807514d4c2125edb8f5055"
server_key_iv: "e8957a5860482e66aecc8a19326ba13d"
server_key_version: "encryptionkey@0"
I20260810 06:25:04.235854 6631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 06:25:04.238739 6658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:04.240370 6631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 06:25:04.240496 6631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "82714ede57e64b70a9a7acb4c2fea98c"
format_stamp: "Formatted at 2026-08-10 06:25:04 on dist-test-slave-45dx"
server_key: "2c704ac520807514d4c2125edb8f5055"
server_key_iv: "e8957a5860482e66aecc8a19326ba13d"
server_key_version: "encryptionkey@0"
I20260810 06:25:04.240602 6631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:04.288293 6631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:04.291913 6631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:04.292114 6631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:04.293851 6631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:04.293939 6631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:04.294045 6631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:04.294094 6631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:04 dist-test-slave-45dx krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:04.308976 6631 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:38147
I20260810 06:25:04.309007 6847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:38147 every 8 connection(s)
I20260810 06:25:04.309974 6631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:04.312119 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.295790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44241 (local address 127.4.73.2:47121)
0810 06:25:04.296327 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.296381 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.297655 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.298374 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.298381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.298888 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:04.299625 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.299648 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.301010 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.301014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.302966 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.302979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.303721 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.303726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.304022 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.304825 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:04.304850 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:04.306994 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:04.309209 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.309215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.309227 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.309508 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.309789 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.309792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.309795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.309961 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.310371 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:04.310379 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:04.310699 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.310912 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.311236 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":170,"threads_started":1}
I20260810 06:25:04.312767 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.294538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:47121 (local address 127.4.73.62:44241)
0810 06:25:04.294669 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.294674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.296507 (+ 1833us) server_negotiation.cc:407] Connection header received
0810 06:25:04.297950 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.297957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.298047 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.298175 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:04.299843 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.300794 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.303090 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.303512 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.303650 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.307168 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:04.307172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:04.307174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:04.307196 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:04.309082 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.309602 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.309605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.309607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.309694 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.310058 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.310062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.310064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.310268 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:04.310371 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.312454 (+ 2083us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.312591 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 06:25:04.313686 6850 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44241
I20260810 06:25:04.313825 6850 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:04.314375 6850 heartbeater.cc:511] Master 127.4.73.62:44241 requested a full tablet report, sending...
I20260810 06:25:04.315634 6260 ts_manager.cc:194] Registered new tserver with Master: 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147)
I20260810 06:25:04.316512 6260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:47121
I20260810 06:25:04.318182 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6631
I20260810 06:25:04.318292 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:04.318535 4388 external_mini_cluster.cc:1428] Setting key 065a60ef0aaa5f3efee83874f1a57a7f
I20260810 06:25:04.324801 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.316275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:59001 (local address 127.4.73.62:44241)
0810 06:25:04.316418 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.316424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.316645 (+ 221us) server_negotiation.cc:407] Connection header received
0810 06:25:04.316826 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.316830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.316877 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.316946 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:04.317425 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.317970 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.319119 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.319380 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.319461 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.320667 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:04.320673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:04.320677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:04.320715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:04.322850 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.323403 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.323406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.323407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.323495 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.323830 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.323832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.323833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.324038 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:04.324116 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.324481 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.324626 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 06:25:04.324869 6859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.316175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44241 (local address 127.4.73.2:59001)
0810 06:25:04.316563 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.316579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.316706 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.316997 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.317000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.317186 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:04.317289 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.317294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.318134 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.318137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.318984 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.318996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.319495 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.319499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.319591 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.320157 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:04.320168 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:04.320532 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:04.323006 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.323012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.323015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.323298 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.323592 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.323597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.323599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.323726 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.324133 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:04.324139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:04.324258 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.324410 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.324587 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:04.349143 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44241
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:04.477332 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:04.477638 6863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:04.477705 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:04.482996 6863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:04.483070 6863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:04.483112 6863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:04.483179 6863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:04.489279 6863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:44241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:04.490549 6863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:04.491729 6863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:04.503189 6875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:04.503214 6871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:04.503152 6873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:04.503760 6863 server_base.cc:1034] running on GCE node
I20260810 06:25:04.504185 6863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:04.504891 6863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:04.506067 6863 hybrid_clock.cc:648] HybridClock initialized: now 1786343104506017 us; error 56 us; skew 500 ppm
Aug 10 06:25:04 dist-test-slave-45dx krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:04.508941 6863 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:04.509864 6863 webserver.cc:460] Webserver started at http://127.4.73.3:44745/ using document root <none> and password file <none>
I20260810 06:25:04.510442 6863 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:04.510507 6863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:04.510757 6863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:04.513002 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3"
format_stamp: "Formatted at 2026-08-10 06:25:04 on dist-test-slave-45dx"
server_key: "a7403f8ffdf9b436faf196816a9a9f9a"
server_key_iv: "18dac4944001b5ada102184e3af9f535"
server_key_version: "encryptionkey@0"
I20260810 06:25:04.513629 6863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3"
format_stamp: "Formatted at 2026-08-10 06:25:04 on dist-test-slave-45dx"
server_key: "a7403f8ffdf9b436faf196816a9a9f9a"
server_key_iv: "18dac4944001b5ada102184e3af9f535"
server_key_version: "encryptionkey@0"
I20260810 06:25:04.517572 6863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:25:04.520093 6887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:04.521442 6863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:04.521554 6863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3"
format_stamp: "Formatted at 2026-08-10 06:25:04 on dist-test-slave-45dx"
server_key: "a7403f8ffdf9b436faf196816a9a9f9a"
server_key_iv: "18dac4944001b5ada102184e3af9f535"
server_key_version: "encryptionkey@0"
I20260810 06:25:04.521656 6863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:04.550607 6863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:04.553817 6863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:04.554049 6863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:04.555969 6863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:04.556031 6863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:04.556095 6863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:04.556129 6863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:04 dist-test-slave-45dx krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:04.572562 6863 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:36325
I20260810 06:25:04.572643 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.557011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44241 (local address 127.4.73.3:58263)
0810 06:25:04.557500 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.557550 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.558437 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.559041 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.559052 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.559641 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:04.560626 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.560670 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.561573 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.561577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.562680 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.562692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.563494 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.563499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.563729 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.564514 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:04.564536 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:04.567086 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:04.569340 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.569345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.569356 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.569668 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.569978 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.569981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.569983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.570168 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.570590 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:04.570598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:04.570930 (+ 332us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.571173 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.571549 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":157,"threads_started":1}
I20260810 06:25:04.573082 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.556530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:58263 (local address 127.4.73.62:44241)
0810 06:25:04.556698 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.556703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.557649 (+ 946us) server_negotiation.cc:407] Connection header received
0810 06:25:04.558672 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.558676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.558737 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.558826 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:04.560817 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.561439 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.562834 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.563279 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.563449 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.567266 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:04.567275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:04.567277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:04.567319 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:04.569174 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.569784 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.569788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.569789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.569887 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.570270 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.570274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.570275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.570466 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:04.570591 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.572770 (+ 2179us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.572906 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:25:04.572641 7089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:36325 every 8 connection(s)
I20260810 06:25:04.573706 6863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:04.575322 7091 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44241
I20260810 06:25:04.575461 7091 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:04.575963 7091 heartbeater.cc:511] Master 127.4.73.62:44241 requested a full tablet report, sending...
I20260810 06:25:04.576568 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6863
I20260810 06:25:04.576690 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:04.577023 4388 external_mini_cluster.cc:1428] Setting key 8d6a15a5d7d39e1cd0dbbcab40b0b5b0
I20260810 06:25:04.577138 6260 ts_manager.cc:194] Registered new tserver with Master: ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3:36325)
I20260810 06:25:04.577988 6260 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:58263
I20260810 06:25:04.579505 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:25:04.586346 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.576676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:49461 (local address 127.4.73.62:44241)
0810 06:25:04.576816 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.576820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.577308 (+ 488us) server_negotiation.cc:407] Connection header received
0810 06:25:04.577714 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.577717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.577767 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.577854 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:04.578449 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.579203 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.581269 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.581515 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.581618 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.582804 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:04.582808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:04.582811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:04.582837 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:04.584604 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.585138 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.585143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.585146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.585258 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.585555 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.585560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.585562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.585733 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:04.585844 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.586094 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.586199 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260810 06:25:04.586359 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.576686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44241 (local address 127.4.73.3:49461)
0810 06:25:04.577209 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.577242 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.577374 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.577871 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.577874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.578114 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:04.578301 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.578313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.580503 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.580506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.581140 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.581150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.581622 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.581626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.581713 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.582334 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:04.582346 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:04.582656 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:04.584729 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.584732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.584733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.584995 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.585374 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:04.585376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:04.585377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:04.585453 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:04.585835 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:04.585838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:04.585932 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.586039 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.586187 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":335,"threads_started":1}
Aug 10 06:25:04 dist-test-slave-45dx krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:25:04 dist-test-slave-45dx krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:04.611807 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.603050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.4.73.62:44241)
0810 06:25:04.603169 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.603172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.603184 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:25:04.603222 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.603225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.603261 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.603331 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:04.604121 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.604685 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.605705 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.605908 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.605984 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.608685 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:04.608688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:04.608690 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:04.608714 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:04.610167 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.610588 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.610592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.610594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.610679 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:04.610950 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:04.610953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:04.610955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:04.611142 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:04.611283 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.611520 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.611640 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:25:04.614915 6260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:04.618156 6260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:04.633735 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.626874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.4.73.3:36325)
0810 06:25:04.627762 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.627775 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.627801 (+ 26us) server_negotiation.cc:407] Connection header received
0810 06:25:04.628466 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.628474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.628697 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.628889 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:04.629191 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.630335 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.632151 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.632900 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.633012 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.633212 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.633425 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.633530 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":144,"threads_started":1}
I20260810 06:25:04.633785 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.628088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40319 (local address 127.0.0.1:38654)
0810 06:25:04.628507 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.628520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.628597 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.629243 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.629246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.629262 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:04.629354 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.629361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.630678 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.630684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.632123 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.632134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.633352 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.633356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.633440 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.633462 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.633558 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.633647 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":100,"thread_start_us":110,"threads_started":1}
I20260810 06:25:04.633809 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.626757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:36325 (local address 127.0.0.1:55450)
0810 06:25:04.627304 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.627397 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.628068 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.628869 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.628875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.628898 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:04.629057 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.629067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.630638 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.630642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.631994 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.632009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.633026 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.633029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.633238 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.633279 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.633396 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.633527 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":1,"thread_start_us":140,"threads_started":1}
I20260810 06:25:04.634166 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.628268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.4.73.1:40319)
0810 06:25:04.628714 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.628719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.628737 (+ 18us) server_negotiation.cc:407] Connection header received
0810 06:25:04.628817 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.628823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.629050 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.629202 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:04.629520 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.630531 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.632263 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.633068 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.633166 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.633290 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.633948 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.634046 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":185,"threads_started":1}
I20260810 06:25:04.634809 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.626962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:38147 (local address 127.0.0.1:49378)
0810 06:25:04.628092 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.628106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.628213 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.629737 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.629740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.629760 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:04.629854 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.629861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.631510 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.631514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.632595 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.632605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.633635 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.633638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.634552 (+ 914us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.634574 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.634628 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.634688 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"thread_start_us":92,"threads_started":1}
I20260810 06:25:04.635423 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.627664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.4.73.2:38147)
0810 06:25:04.629286 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.629301 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.629321 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:25:04.629391 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.629396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.629582 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.629739 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:04.629989 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.631361 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.632729 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.633472 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.635009 (+ 1537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.635167 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.635217 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.635293 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1517,"thread_start_us":92,"threads_started":1}
I20260810 06:25:04.636940 6973 tablet_service.cc:1467] Processing CreateTablet for tablet 716c848db7484660b211b743a4dcd053 (DEFAULT_TABLE table=test-table [id=8840cb4a25c74dbd9784e12ee0a948c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:04.636973 6475 tablet_service.cc:1467] Processing CreateTablet for tablet 716c848db7484660b211b743a4dcd053 (DEFAULT_TABLE table=test-table [id=8840cb4a25c74dbd9784e12ee0a948c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:04.637949 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716c848db7484660b211b743a4dcd053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:04.638055 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716c848db7484660b211b743a4dcd053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:04.645296 7126 tablet_bootstrap.cc:492] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Bootstrap starting.
I20260810 06:25:04.645200 6735 tablet_service.cc:1467] Processing CreateTablet for tablet 716c848db7484660b211b743a4dcd053 (DEFAULT_TABLE table=test-table [id=8840cb4a25c74dbd9784e12ee0a948c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:04.646333 6735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716c848db7484660b211b743a4dcd053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:04.647089 7127 tablet_bootstrap.cc:492] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Bootstrap starting.
I20260810 06:25:04.648838 7126 tablet_bootstrap.cc:654] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:04.649466 7127 tablet_bootstrap.cc:654] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:04.649926 7126 log.cc:824] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:04.650362 7127 log.cc:824] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:04.652120 7126 tablet_bootstrap.cc:492] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: No bootstrap required, opened a new log
I20260810 06:25:04.652355 7126 ts_tablet_manager.cc:1397] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:25:04.654402 7132 tablet_bootstrap.cc:492] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Bootstrap starting.
I20260810 06:25:04.656989 7132 tablet_bootstrap.cc:654] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:04.657011 7127 tablet_bootstrap.cc:492] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: No bootstrap required, opened a new log
I20260810 06:25:04.657292 7127 ts_tablet_manager.cc:1397] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20260810 06:25:04.657277 7126 raft_consensus.cc:348] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.657641 7126 raft_consensus.cc:374] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:04.657719 7126 raft_consensus.cc:729] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f686ba1dda43fb800e3d199d89cb4b, State: Initialized, Role: FOLLOWER
I20260810 06:25:04.657912 7132 log.cc:824] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:04.658229 7126 consensus_queue.cc:260] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.659397 7126 ts_tablet_manager.cc:1428] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:25:04.659529 6614 heartbeater.cc:503] Master 127.4.73.62:44241 was elected leader, sending a full tablet report...
I20260810 06:25:04.659925 7132 tablet_bootstrap.cc:492] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: No bootstrap required, opened a new log
I20260810 06:25:04.660104 7132 ts_tablet_manager.cc:1397] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:25:04.661834 7127 raft_consensus.cc:348] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.662091 7127 raft_consensus.cc:374] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:04.662144 7127 raft_consensus.cc:729] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed05f1efc0ad4c8fb1f9b07f564252d3, State: Initialized, Role: FOLLOWER
I20260810 06:25:04.662648 7127 consensus_queue.cc:260] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.663830 7127 ts_tablet_manager.cc:1428] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260810 06:25:04.664199 7091 heartbeater.cc:503] Master 127.4.73.62:44241 was elected leader, sending a full tablet report...
I20260810 06:25:04.664386 7132 raft_consensus.cc:348] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.664674 7132 raft_consensus.cc:374] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:04.664741 7132 raft_consensus.cc:729] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82714ede57e64b70a9a7acb4c2fea98c, State: Initialized, Role: FOLLOWER
I20260810 06:25:04.665185 7132 consensus_queue.cc:260] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.665957 6850 heartbeater.cc:503] Master 127.4.73.62:44241 was elected leader, sending a full tablet report...
I20260810 06:25:04.666340 7132 ts_tablet_manager.cc:1428] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 06:25:04.787792 7138 raft_consensus.cc:482] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:04.788038 7138 raft_consensus.cc:504] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.789296 7138 leader_election.cc:288] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147), b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319)
W20260810 06:25:04.789652 6615 tablet.cc:2369] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:04.793455 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.789725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:51015 (local address 127.4.73.2:38147)
0810 06:25:04.789878 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.789882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.789902 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:25:04.789967 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.789970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.790021 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.790113 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:04.790472 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.791254 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.792463 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.792854 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.792933 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.792985 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.793295 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.793355 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 06:25:04.793519 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.789628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:38147 (local address 127.4.73.3:51015)
0810 06:25:04.789765 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.789782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.789885 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.790161 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.790164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.790188 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:04.790340 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.790349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.791382 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.791384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.792331 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.792341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.792997 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.793000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.793113 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.793164 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.793271 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.793390 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 06:25:04.794232 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716c848db7484660b211b743a4dcd053" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82714ede57e64b70a9a7acb4c2fea98c" is_pre_election: true
I20260810 06:25:04.794644 6775 raft_consensus.cc:2454] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 0.
I20260810 06:25:04.795246 6891 leader_election.cc:302] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82714ede57e64b70a9a7acb4c2fea98c, ed05f1efc0ad4c8fb1f9b07f564252d3; no voters:
I20260810 06:25:04.795674 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.789934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40319 (local address 127.4.73.3:41151)
0810 06:25:04.790496 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:04.790511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:04.790599 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:04.790813 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:04.790815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:04.790829 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:04.790936 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.790941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.791761 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.791764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.794290 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:04.794301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.795217 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.795221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:04.795334 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.795356 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:04.795462 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 06:25:04.795537 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":177,"threads_started":1}
I20260810 06:25:04.795637 7138 raft_consensus.cc:2781] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:04.795735 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.790454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:41151 (local address 127.4.73.1:40319)
0810 06:25:04.790582 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.790586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.790600 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:04.790672 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.790675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.790722 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.790797 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:04.791053 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.791654 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.794516 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.795082 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.795207 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.795269 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.795530 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.795615 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:25:04.795897 7138 raft_consensus.cc:482] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:04.795976 7138 raft_consensus.cc:3037] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:04.796286 6534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716c848db7484660b211b743a4dcd053" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" is_pre_election: true
I20260810 06:25:04.796634 6534 raft_consensus.cc:2454] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 0.
I20260810 06:25:04.797715 7138 raft_consensus.cc:504] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.798182 7138 leader_election.cc:288] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 election: Requested vote from peers 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147), b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319)
I20260810 06:25:04.798528 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716c848db7484660b211b743a4dcd053" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82714ede57e64b70a9a7acb4c2fea98c"
I20260810 06:25:04.798682 6775 raft_consensus.cc:3037] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:04.798686 6534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716c848db7484660b211b743a4dcd053" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f686ba1dda43fb800e3d199d89cb4b"
I20260810 06:25:04.798822 6534 raft_consensus.cc:3037] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:04.799808 6775 raft_consensus.cc:2454] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 1.
I20260810 06:25:04.800143 6891 leader_election.cc:302] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82714ede57e64b70a9a7acb4c2fea98c, ed05f1efc0ad4c8fb1f9b07f564252d3; no voters:
I20260810 06:25:04.800593 6534 raft_consensus.cc:2454] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 1.
I20260810 06:25:04.800879 7138 raft_consensus.cc:2781] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:04.801177 7138 raft_consensus.cc:686] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 LEADER]: Becoming Leader. State: Replica: ed05f1efc0ad4c8fb1f9b07f564252d3, State: Running, Role: LEADER
I20260810 06:25:04.801534 7138 consensus_queue.cc:237] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } }
I20260810 06:25:04.805910 6260 catalog_manager.cc:5495] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } health_report { overall_health: UNKNOWN } } }
W20260810 06:25:04.811803 6851 tablet.cc:2369] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:25:04.825011 7092 tablet.cc:2369] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:04.832548 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:04.827891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.4.73.3:36325)
0810 06:25:04.828047 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:04.828051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:04.828068 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:25:04.828292 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:04.828295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:04.828393 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:04.828488 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:04.828846 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.829382 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.830917 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:04.831118 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:04.831193 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:04.831631 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:04.832038 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:04.832137 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:04.832272 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 06:25:04.832391 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 06:25:04.839644 6775 raft_consensus.cc:1261] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Refusing update from remote peer ed05f1efc0ad4c8fb1f9b07f564252d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:04.840543 7138 consensus_queue.cc:1035] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:04.840961 6534 raft_consensus.cc:1261] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Refusing update from remote peer ed05f1efc0ad4c8fb1f9b07f564252d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:04.841476 7138 consensus_queue.cc:1035] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:04.848727 7150 mvcc.cc:205] Tried to move back new op lower bound from 7316861357414354944 to 7316861357271498752. Current Snapshot: MvccSnapshot[applied={T|T < 7316861357414354944}]
I20260810 06:25:04.863317 6260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:04.863587 6260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:04.866122 6260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5: Sending DeleteTablet for 3 replicas of tablet 716c848db7484660b211b743a4dcd053
I20260810 06:25:04.867079 6475 tablet_service.cc:1514] Processing DeleteTablet for tablet 716c848db7484660b211b743a4dcd053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:04 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38654
I20260810 06:25:04.867166 6735 tablet_service.cc:1514] Processing DeleteTablet for tablet 716c848db7484660b211b743a4dcd053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:04 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:49378
I20260810 06:25:04.867640 7173 tablet_replica.cc:330] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: stopping tablet replica
I20260810 06:25:04.867671 6973 tablet_service.cc:1514] Processing DeleteTablet for tablet 716c848db7484660b211b743a4dcd053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:04 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:55450
I20260810 06:25:04.867944 7173 raft_consensus.cc:2229] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:04.867967 6260 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:25:04.868125 7170 tablet_replica.cc:330] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: stopping tablet replica
I20260810 06:25:04.868343 7173 raft_consensus.cc:2258] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 06:25:04.868376 6260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:04.868434 7170 raft_consensus.cc:2229] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:04.868873 7170 raft_consensus.cc:2258] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:04.869976 7173 ts_tablet_manager.cc:1905] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:04.870299 7170 ts_tablet_manager.cc:1905] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:04.872921 7173 ts_tablet_manager.cc:1918] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:04.873020 7173 log.cc:1196] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/716c848db7484660b211b743a4dcd053
I20260810 06:25:04.873347 7173 ts_tablet_manager.cc:1939] T 716c848db7484660b211b743a4dcd053 P 82714ede57e64b70a9a7acb4c2fea98c: Deleting consensus metadata
I20260810 06:25:04.873639 6475 tablet_service.cc:1467] Processing CreateTablet for tablet f62389f3a36d4ea181f7196802e25015 (DEFAULT_TABLE table=test-table [id=bbcba5a51fd84be8a7b59fdd7191fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:04.873922 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62389f3a36d4ea181f7196802e25015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:04.873903 6973 tablet_service.cc:1467] Processing CreateTablet for tablet f62389f3a36d4ea181f7196802e25015 (DEFAULT_TABLE table=test-table [id=bbcba5a51fd84be8a7b59fdd7191fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:04.874138 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62389f3a36d4ea181f7196802e25015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:04.876245 7126 tablet_bootstrap.cc:492] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: Bootstrap starting.
I20260810 06:25:04.876896 7127 tablet_bootstrap.cc:492] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: Bootstrap starting.
I20260810 06:25:04.878044 7126 tablet_bootstrap.cc:654] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:04.878660 7127 tablet_bootstrap.cc:654] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:04.879108 6232 catalog_manager.cc:4841] TS 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147): tablet 716c848db7484660b211b743a4dcd053 (table test-table [id=8840cb4a25c74dbd9784e12ee0a948c2]) successfully deleted
I20260810 06:25:04.879778 7170 ts_tablet_manager.cc:1918] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:04.879927 7170 log.cc:1196] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/716c848db7484660b211b743a4dcd053
I20260810 06:25:04.879980 6735 tablet_service.cc:1467] Processing CreateTablet for tablet f62389f3a36d4ea181f7196802e25015 (DEFAULT_TABLE table=test-table [id=bbcba5a51fd84be8a7b59fdd7191fc24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:04.880282 6735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f62389f3a36d4ea181f7196802e25015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:04.880411 7170 ts_tablet_manager.cc:1939] T 716c848db7484660b211b743a4dcd053 P b7f686ba1dda43fb800e3d199d89cb4b: Deleting consensus metadata
I20260810 06:25:04.881670 6232 catalog_manager.cc:4841] TS b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319): tablet 716c848db7484660b211b743a4dcd053 (table test-table [id=8840cb4a25c74dbd9784e12ee0a948c2]) successfully deleted
I20260810 06:25:04.882620 7127 tablet_bootstrap.cc:492] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: No bootstrap required, opened a new log
I20260810 06:25:04.882751 7127 ts_tablet_manager.cc:1397] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260810 06:25:04.882900 7132 tablet_bootstrap.cc:492] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: Bootstrap starting.
I20260810 06:25:04.883450 7127 raft_consensus.cc:348] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:04.883594 7127 raft_consensus.cc:374] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:04.883652 7127 raft_consensus.cc:729] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed05f1efc0ad4c8fb1f9b07f564252d3, State: Initialized, Role: FOLLOWER
I20260810 06:25:04.883836 7127 consensus_queue.cc:260] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:04.884194 7127 ts_tablet_manager.cc:1428] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:04.884310 7132 tablet_bootstrap.cc:654] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:04.884639 7175 tablet_replica.cc:330] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: stopping tablet replica
I20260810 06:25:04.884980 7175 raft_consensus.cc:2229] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:04.885221 7126 tablet_bootstrap.cc:492] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: No bootstrap required, opened a new log
I20260810 06:25:04.885298 7126 ts_tablet_manager.cc:1397] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260810 06:25:04.885345 7175 raft_consensus.cc:2258] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:04.886341 7126 raft_consensus.cc:348] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:04.886458 7126 raft_consensus.cc:374] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:04.886511 7126 raft_consensus.cc:729] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f686ba1dda43fb800e3d199d89cb4b, State: Initialized, Role: FOLLOWER
I20260810 06:25:04.886646 7126 consensus_queue.cc:260] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:04.886926 7126 ts_tablet_manager.cc:1428] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:04.887948 7175 ts_tablet_manager.cc:1905] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:04.890545 7175 ts_tablet_manager.cc:1918] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:04.890666 7175 log.cc:1196] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/716c848db7484660b211b743a4dcd053
I20260810 06:25:04.890973 7175 ts_tablet_manager.cc:1939] T 716c848db7484660b211b743a4dcd053 P ed05f1efc0ad4c8fb1f9b07f564252d3: Deleting consensus metadata
I20260810 06:25:04.891984 6234 catalog_manager.cc:4841] TS ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3:36325): tablet 716c848db7484660b211b743a4dcd053 (table test-table [id=8840cb4a25c74dbd9784e12ee0a948c2]) successfully deleted
I20260810 06:25:04.892978 7132 tablet_bootstrap.cc:492] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: No bootstrap required, opened a new log
I20260810 06:25:04.893095 7132 ts_tablet_manager.cc:1397] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20260810 06:25:04.893615 7132 raft_consensus.cc:348] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:04.893728 7132 raft_consensus.cc:374] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:04.893769 7132 raft_consensus.cc:729] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82714ede57e64b70a9a7acb4c2fea98c, State: Initialized, Role: FOLLOWER
I20260810 06:25:04.893939 7132 consensus_queue.cc:260] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:04.894208 7132 ts_tablet_manager.cc:1428] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:25:05.010536 7139 raft_consensus.cc:482] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:05.010704 7139 raft_consensus.cc:504] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.012120 7139 leader_election.cc:288] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319), ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3:36325)
I20260810 06:25:05.014883 7137 raft_consensus.cc:482] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:05.015002 7137 raft_consensus.cc:504] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.015992 7137 leader_election.cc:288] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147), ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3:36325)
I20260810 06:25:05.017854 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.012528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:49759 (local address 127.4.73.1:40319)
0810 06:25:05.012665 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.012670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.012688 (+ 18us) server_negotiation.cc:407] Connection header received
0810 06:25:05.012836 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.012839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.012888 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.012976 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:05.013470 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.014225 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.016354 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.017041 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.017124 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.017185 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.017609 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.017709 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:25:05.018014 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.013232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:55175 (local address 127.4.73.3:36325)
0810 06:25:05.013347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.013351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.013363 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:25:05.013414 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.013416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.013451 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.013534 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:05.013796 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.014323 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.016465 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.017049 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.017124 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.017186 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.017822 (+ 636us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.017901 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 06:25:05.018129 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.012702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:36325 (local address 127.4.73.2:55175)
0810 06:25:05.013030 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.013044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.013141 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.013546 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.013549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.013564 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:05.013688 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.013693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.014429 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.014432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.016133 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.016143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.017530 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.017536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.017687 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.017709 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.017785 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.017863 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":160,"spinlock_wait_cycles":3968,"thread_start_us":144,"threads_started":1}
I20260810 06:25:05.018560 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.012306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40319 (local address 127.4.73.2:49759)
0810 06:25:05.012602 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.012617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.012725 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.013102 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.013106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.013143 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:05.013319 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.013327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.014425 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.014428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.016131 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.016143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.017342 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.017346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.017463 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.017491 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.018017 (+ 526us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.018133 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":125,"spinlock_wait_cycles":6656,"thread_start_us":86,"threads_started":1}
I20260810 06:25:05.018832 7011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62389f3a36d4ea181f7196802e25015" candidate_uuid: "82714ede57e64b70a9a7acb4c2fea98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" is_pre_election: true
I20260810 06:25:05.019007 6534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62389f3a36d4ea181f7196802e25015" candidate_uuid: "82714ede57e64b70a9a7acb4c2fea98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" is_pre_election: true
I20260810 06:25:05.019037 7011 raft_consensus.cc:2454] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82714ede57e64b70a9a7acb4c2fea98c in term 0.
I20260810 06:25:05.019143 6534 raft_consensus.cc:2454] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82714ede57e64b70a9a7acb4c2fea98c in term 0.
I20260810 06:25:05.019629 6664 leader_election.cc:302] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82714ede57e64b70a9a7acb4c2fea98c, b7f686ba1dda43fb800e3d199d89cb4b; no voters:
I20260810 06:25:05.020009 7139 raft_consensus.cc:2781] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:05.020077 7139 raft_consensus.cc:482] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:05.020117 7139 raft_consensus.cc:3037] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:05.021775 7139 raft_consensus.cc:504] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.022104 7139 leader_election.cc:288] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [CANDIDATE]: Term 1 election: Requested vote from peers b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319), ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3:36325)
I20260810 06:25:05.022444 6534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62389f3a36d4ea181f7196802e25015" candidate_uuid: "82714ede57e64b70a9a7acb4c2fea98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f686ba1dda43fb800e3d199d89cb4b"
I20260810 06:25:05.022550 6534 raft_consensus.cc:3037] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:05.023766 6534 raft_consensus.cc:2454] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82714ede57e64b70a9a7acb4c2fea98c in term 1.
I20260810 06:25:05.024006 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.017338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:44427 (local address 127.4.73.2:38147)
0810 06:25:05.017442 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.017446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.018892 (+ 1446us) server_negotiation.cc:407] Connection header received
0810 06:25:05.019071 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.019074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.019117 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.019165 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:05.019717 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.020787 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.022711 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.023261 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.023331 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.023373 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.023788 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.023863 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:25:05.024119 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.016989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:38147 (local address 127.4.73.1:44427)
0810 06:25:05.018809 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.018822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.018957 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.019291 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.019293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.019329 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:05.019549 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.019558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.021164 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.021167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.022555 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.022578 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.023504 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.023507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.023633 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.023649 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.023758 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.023829 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"thread_start_us":101,"threads_started":1}
I20260810 06:25:05.024430 7011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62389f3a36d4ea181f7196802e25015" candidate_uuid: "82714ede57e64b70a9a7acb4c2fea98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3"
I20260810 06:25:05.024498 6664 leader_election.cc:302] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82714ede57e64b70a9a7acb4c2fea98c, b7f686ba1dda43fb800e3d199d89cb4b; no voters:
I20260810 06:25:05.024552 7011 raft_consensus.cc:3037] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:05.024669 7139 raft_consensus.cc:2781] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:05.024760 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62389f3a36d4ea181f7196802e25015" candidate_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82714ede57e64b70a9a7acb4c2fea98c" is_pre_election: true
I20260810 06:25:05.024955 7139 raft_consensus.cc:686] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 LEADER]: Becoming Leader. State: Replica: 82714ede57e64b70a9a7acb4c2fea98c, State: Running, Role: LEADER
I20260810 06:25:05.025209 7139 consensus_queue.cc:237] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.025905 7011 raft_consensus.cc:2454] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82714ede57e64b70a9a7acb4c2fea98c in term 1.
I20260810 06:25:05.028133 6260 catalog_manager.cc:5495] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c reported cstate change: term changed from 0 to 1, leader changed from <none> to 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2). New cstate: current_term: 1 leader_uuid: "82714ede57e64b70a9a7acb4c2fea98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } health_report { overall_health: UNKNOWN } } }
I20260810 06:25:05.029317 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.016574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:60837 (local address 127.4.73.3:36325)
0810 06:25:05.017312 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.017317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.019720 (+ 2403us) server_negotiation.cc:407] Connection header received
0810 06:25:05.019826 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.019829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.019874 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.019959 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:05.024716 (+ 4757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.025664 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.027255 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.027978 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.028126 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.028219 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.028979 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.029089 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":72,"threads_started":1}
I20260810 06:25:05.028859 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.016255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:36325 (local address 127.4.73.1:60837)
0810 06:25:05.018323 (+ 2068us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.018338 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.018457 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.022897 (+ 4440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.022901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.022920 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:05.023032 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.023038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.025799 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.025813 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.027112 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.027123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.028101 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.028105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.028387 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.028423 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.028518 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.028612 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1993,"thread_start_us":84,"threads_started":1}
I20260810 06:25:05.030498 7011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f62389f3a36d4ea181f7196802e25015" candidate_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" is_pre_election: true
I20260810 06:25:05.030665 7011 raft_consensus.cc:2379] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7f686ba1dda43fb800e3d199d89cb4b in current term 1: Already voted for candidate 82714ede57e64b70a9a7acb4c2fea98c in this term.
I20260810 06:25:05.031284 6424 leader_election.cc:302] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7f686ba1dda43fb800e3d199d89cb4b; no voters: 82714ede57e64b70a9a7acb4c2fea98c, ed05f1efc0ad4c8fb1f9b07f564252d3
I20260810 06:25:05.031602 7137 raft_consensus.cc:2726] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Aug 10 06:25:05 dist-test-slave-45dx krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:05.078238 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.066567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44241 (local address 127.0.0.1:35800)
0810 06:25:05.067002 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.067020 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.067148 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.067389 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.067392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.067946 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:05.068107 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.068116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.069309 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.069313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.070386 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.070401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.070838 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.070841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.070917 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.071735 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:05.071754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:05.073891 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:05.076236 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:05.076248 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:05.076254 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:05.076553 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:05.076887 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:05.076891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:05.076893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:05.077052 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:05.077530 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:05.077537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:05.077672 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.077825 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.077994 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":179,"threads_started":1}
I20260810 06:25:05.078236 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.066592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.4.73.62:44241)
0810 06:25:05.066726 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.066730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.067089 (+ 359us) server_negotiation.cc:407] Connection header received
0810 06:25:05.067242 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.067245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.067290 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.067369 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:05.068248 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.069165 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.070500 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.070728 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.070834 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.074039 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:05.074042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:05.074044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:05.074063 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:05.076126 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:05.076663 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:05.076668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:05.076672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:05.076760 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:05.077164 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:05.077166 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:05.077168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:05.077424 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:05.077522 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.077871 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.078071 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":55}
I20260810 06:25:05.082779 6260 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:35800:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 06:25:05.083521 6260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:05.088894 6475 tablet_service.cc:1467] Processing CreateTablet for tablet 59270c381e6342238032cf08709f5e1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05d4eb047f404337a1e610cd334fb53a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:05.089068 6735 tablet_service.cc:1467] Processing CreateTablet for tablet 59270c381e6342238032cf08709f5e1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05d4eb047f404337a1e610cd334fb53a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:05.089445 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59270c381e6342238032cf08709f5e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:05.089474 6735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59270c381e6342238032cf08709f5e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:05.089494 6973 tablet_service.cc:1467] Processing CreateTablet for tablet 59270c381e6342238032cf08709f5e1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05d4eb047f404337a1e610cd334fb53a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:05.089843 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59270c381e6342238032cf08709f5e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:05.093566 7132 tablet_bootstrap.cc:492] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: Bootstrap starting.
I20260810 06:25:05.094715 7132 tablet_bootstrap.cc:654] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:05.094907 7127 tablet_bootstrap.cc:492] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: Bootstrap starting.
I20260810 06:25:05.095877 7126 tablet_bootstrap.cc:492] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: Bootstrap starting.
I20260810 06:25:05.096311 7127 tablet_bootstrap.cc:654] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:05.096727 7132 tablet_bootstrap.cc:492] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: No bootstrap required, opened a new log
I20260810 06:25:05.096819 7132 ts_tablet_manager.cc:1397] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260810 06:25:05.097196 7126 tablet_bootstrap.cc:654] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:05.097252 7132 raft_consensus.cc:348] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.097357 7132 raft_consensus.cc:374] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:05.097384 7132 raft_consensus.cc:729] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82714ede57e64b70a9a7acb4c2fea98c, State: Initialized, Role: FOLLOWER
I20260810 06:25:05.097497 7132 consensus_queue.cc:260] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.097860 7187 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.097987 7187 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:05.098704 7127 tablet_bootstrap.cc:492] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: No bootstrap required, opened a new log
I20260810 06:25:05.098800 7127 ts_tablet_manager.cc:1397] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 06:25:05.098894 7132 ts_tablet_manager.cc:1428] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:05.099390 7127 raft_consensus.cc:348] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.099548 7127 raft_consensus.cc:374] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:05.099592 7127 raft_consensus.cc:729] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed05f1efc0ad4c8fb1f9b07f564252d3, State: Initialized, Role: FOLLOWER
I20260810 06:25:05.099725 7127 consensus_queue.cc:260] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.100023 7127 ts_tablet_manager.cc:1428] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 06:25:05.100044 7164 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.100229 7164 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:05.103008 7126 tablet_bootstrap.cc:492] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: No bootstrap required, opened a new log
I20260810 06:25:05.103837 7126 ts_tablet_manager.cc:1397] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260810 06:25:05.104635 7126 raft_consensus.cc:348] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.104771 7126 raft_consensus.cc:374] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:05.104815 7126 raft_consensus.cc:729] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f686ba1dda43fb800e3d199d89cb4b, State: Initialized, Role: FOLLOWER
I20260810 06:25:05.104988 7126 consensus_queue.cc:260] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.105257 7137 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.105403 7137 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:05.105803 7126 ts_tablet_manager.cc:1428] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:05.418947 7164 raft_consensus.cc:482] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:05.419121 7164 raft_consensus.cc:504] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.419662 7164 leader_election.cc:288] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319), 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147)
I20260810 06:25:05.420148 6534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59270c381e6342238032cf08709f5e1d" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" is_pre_election: true
I20260810 06:25:05.420182 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59270c381e6342238032cf08709f5e1d" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82714ede57e64b70a9a7acb4c2fea98c" is_pre_election: true
I20260810 06:25:05.420332 6534 raft_consensus.cc:2454] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 0.
I20260810 06:25:05.420331 6775 raft_consensus.cc:2454] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 0.
I20260810 06:25:05.420653 6891 leader_election.cc:302] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7f686ba1dda43fb800e3d199d89cb4b, ed05f1efc0ad4c8fb1f9b07f564252d3; no voters:
I20260810 06:25:05.420892 7164 raft_consensus.cc:2781] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:05.420989 7164 raft_consensus.cc:482] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:05.421041 7164 raft_consensus.cc:3037] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:05.422567 7164 raft_consensus.cc:504] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.422986 7164 leader_election.cc:288] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 election: Requested vote from peers b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319), 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147)
I20260810 06:25:05.423427 6775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59270c381e6342238032cf08709f5e1d" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82714ede57e64b70a9a7acb4c2fea98c"
I20260810 06:25:05.423447 6534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59270c381e6342238032cf08709f5e1d" candidate_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7f686ba1dda43fb800e3d199d89cb4b"
I20260810 06:25:05.423552 6534 raft_consensus.cc:3037] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:05.423552 6775 raft_consensus.cc:3037] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:05.424705 6534 raft_consensus.cc:2454] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 1.
I20260810 06:25:05.424705 6775 raft_consensus.cc:2454] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed05f1efc0ad4c8fb1f9b07f564252d3 in term 1.
I20260810 06:25:05.424991 6891 leader_election.cc:302] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82714ede57e64b70a9a7acb4c2fea98c, ed05f1efc0ad4c8fb1f9b07f564252d3; no voters:
I20260810 06:25:05.425166 7164 raft_consensus.cc:2781] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:05.425376 7164 raft_consensus.cc:686] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 LEADER]: Becoming Leader. State: Replica: ed05f1efc0ad4c8fb1f9b07f564252d3, State: Running, Role: LEADER
I20260810 06:25:05.425580 7164 consensus_queue.cc:237] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } }
I20260810 06:25:05.426374 7216 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: TxnStatusTablet state changed. Reason: New leader ed05f1efc0ad4c8fb1f9b07f564252d3. Latest consensus state: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.426479 7216 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:05.426786 7219 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 06:25:05.426937 7219 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 06:25:05.427510 6260 catalog_manager.cc:5495] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } health_report { overall_health: HEALTHY } } }
I20260810 06:25:05.469764 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.465596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55464 (local address 127.4.73.3:36325)
0810 06:25:05.465768 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.465775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.466029 (+ 254us) server_negotiation.cc:407] Connection header received
0810 06:25:05.466208 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.466214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.466275 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.466382 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:05.466783 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.467551 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.468421 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.468760 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.468839 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.469277 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:05.469382 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:05.469449 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.469560 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.469626 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:25:05.469764 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.465489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:36325 (local address 127.0.0.1:55464)
0810 06:25:05.465928 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.465944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.466081 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.466529 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.466533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.466542 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:05.466653 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.466659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.467655 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.467659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.468266 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.468274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.468952 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.468954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.469059 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.469135 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:05.469453 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:05.469469 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.469550 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.469649 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325}
I20260810 06:25:05.472946 6534 raft_consensus.cc:1261] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Refusing update from remote peer ed05f1efc0ad4c8fb1f9b07f564252d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:05.472942 6775 raft_consensus.cc:1261] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Refusing update from remote peer ed05f1efc0ad4c8fb1f9b07f564252d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:05.473417 7216 consensus_queue.cc:1035] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:05.473594 7164 consensus_queue.cc:1035] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:05.475046 7187 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: TxnStatusTablet state changed. Reason: New leader ed05f1efc0ad4c8fb1f9b07f564252d3. Latest consensus state: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.475164 7187 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:05.475131 7137 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: TxnStatusTablet state changed. Reason: New leader ed05f1efc0ad4c8fb1f9b07f564252d3. Latest consensus state: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.475263 7137 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:05.478958 7187 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.479068 7187 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P 82714ede57e64b70a9a7acb4c2fea98c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:05.479439 7226 mvcc.cc:205] Tried to move back new op lower bound from 7316861360012591104 to 7316861359825113088. Current Snapshot: MvccSnapshot[applied={T|T < 7316861360012591104}]
I20260810 06:25:05.479713 7225 mvcc.cc:205] Tried to move back new op lower bound from 7316861360012591104 to 7316861359825113088. Current Snapshot: MvccSnapshot[applied={T|T < 7316861360012591104}]
I20260810 06:25:05.480058 7137 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.480160 7137 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P b7f686ba1dda43fb800e3d199d89cb4b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:05.480139 7216 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.480244 7216 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:05.481146 7164 tablet_replica.cc:439] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 } } peers { permanent_uuid: "82714ede57e64b70a9a7acb4c2fea98c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 38147 } } peers { permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 } } }
I20260810 06:25:05.481266 7164 tablet_replica.cc:442] T 59270c381e6342238032cf08709f5e1d P ed05f1efc0ad4c8fb1f9b07f564252d3: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:05.486351 7187 consensus_queue.cc:1035] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed05f1efc0ad4c8fb1f9b07f564252d3" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 36325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:05.491492 7187 consensus_queue.cc:1035] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7f686ba1dda43fb800e3d199d89cb4b" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:05.497534 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.492367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.4.73.2:38147)
0810 06:25:05.492484 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.492488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.492776 (+ 288us) server_negotiation.cc:407] Connection header received
0810 06:25:05.493234 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.493238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.493294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.493397 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:05.493669 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.494420 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.495604 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.496066 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.496182 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.496615 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:05.497158 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:05.497268 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.497353 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.497404 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:05.509517 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.505872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:35825 (local address 127.4.73.3:36325)
0810 06:25:05.506028 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.506033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.506047 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:05.506094 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.506098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.506178 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.506259 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:05.506572 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.507326 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.508163 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.508581 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.508704 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.509041 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:05.509156 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:05.509234 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.509330 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.509376 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:25:05.509758 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.505345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:36325 (local address 127.4.73.2:35825)
0810 06:25:05.505714 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.505743 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.505862 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.506380 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.506382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.506388 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:05.506465 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.506469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.507423 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.507429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.508035 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.508043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.508732 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.508738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.508834 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.508915 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:05.509240 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:05.509248 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.509317 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.509400 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20260810 06:25:05.539058 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.534512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:57707 (local address 127.4.73.2:38147)
0810 06:25:05.534799 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.534803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.535033 (+ 230us) server_negotiation.cc:407] Connection header received
0810 06:25:05.535299 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.535305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.535368 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.535418 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:05.535894 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.536568 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.537838 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.538064 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.538138 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.538549 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:05.538629 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:05.538678 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.538849 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.538910 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260810 06:25:05.539510 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.534132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:38147 (local address 127.4.73.3:57707)
0810 06:25:05.534950 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:05.534967 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:05.535123 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:05.535562 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:05.535565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:05.535574 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:05.535689 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.535694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.536663 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.536666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.537710 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:05.537721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.538252 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.538255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:05.538346 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.538394 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:05.538739 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:05.538746 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:25:05.539269 (+ 523us) client_negotiation.cc:240] Negotiation successful
0810 06:25:05.539345 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260810 06:25:05.565946 6260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:05.566121 6260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:05.568490 6260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93f1544b811946ab8610df1bebf2e1f5: Sending DeleteTablet for 3 replicas of tablet f62389f3a36d4ea181f7196802e25015
I20260810 06:25:05.569285 6475 tablet_service.cc:1514] Processing DeleteTablet for tablet f62389f3a36d4ea181f7196802e25015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:05 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38654
I20260810 06:25:05.569880 6735 tablet_service.cc:1514] Processing DeleteTablet for tablet f62389f3a36d4ea181f7196802e25015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:05 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:49378
I20260810 06:25:05.570258 6973 tablet_service.cc:1514] Processing DeleteTablet for tablet f62389f3a36d4ea181f7196802e25015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:05 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:55450
I20260810 06:25:05.570683 7264 tablet_replica.cc:330] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: stopping tablet replica
I20260810 06:25:05.570864 7264 raft_consensus.cc:2229] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:05.570977 7264 raft_consensus.cc:2258] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:05.571635 7264 ts_tablet_manager.cc:1905] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:05.572438 7262 tablet_replica.cc:330] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: stopping tablet replica
I20260810 06:25:05.572611 7262 raft_consensus.cc:2229] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:05.572777 7262 raft_consensus.cc:2258] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:05.573443 7262 ts_tablet_manager.cc:1905] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:05.576749 7263 tablet_replica.cc:330] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: stopping tablet replica
I20260810 06:25:05.576894 7263 raft_consensus.cc:2229] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:05.576930 7262 ts_tablet_manager.cc:1918] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:05.577095 7262 log.cc:1196] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/f62389f3a36d4ea181f7196802e25015
I20260810 06:25:05.577237 7263 raft_consensus.cc:2258] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:05.577533 7262 ts_tablet_manager.cc:1939] T f62389f3a36d4ea181f7196802e25015 P b7f686ba1dda43fb800e3d199d89cb4b: Deleting consensus metadata
I20260810 06:25:05.578444 6232 catalog_manager.cc:4841] TS b7f686ba1dda43fb800e3d199d89cb4b (127.4.73.1:40319): tablet f62389f3a36d4ea181f7196802e25015 (table test-table [id=bbcba5a51fd84be8a7b59fdd7191fc24]) successfully deleted
I20260810 06:25:05.580973 7263 ts_tablet_manager.cc:1905] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:05.584290 7263 ts_tablet_manager.cc:1918] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:05.584409 7263 log.cc:1196] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/f62389f3a36d4ea181f7196802e25015
I20260810 06:25:05.584729 7263 ts_tablet_manager.cc:1939] T f62389f3a36d4ea181f7196802e25015 P 82714ede57e64b70a9a7acb4c2fea98c: Deleting consensus metadata
I20260810 06:25:05.585467 6232 catalog_manager.cc:4841] TS 82714ede57e64b70a9a7acb4c2fea98c (127.4.73.2:38147): tablet f62389f3a36d4ea181f7196802e25015 (table test-table [id=bbcba5a51fd84be8a7b59fdd7191fc24]) successfully deleted
Aug 10 06:25:05 dist-test-slave-45dx krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:05.594305 7264 ts_tablet_manager.cc:1918] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:05.594415 7264 log.cc:1196] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/f62389f3a36d4ea181f7196802e25015
I20260810 06:25:05.594822 7264 ts_tablet_manager.cc:1939] T f62389f3a36d4ea181f7196802e25015 P ed05f1efc0ad4c8fb1f9b07f564252d3: Deleting consensus metadata
I20260810 06:25:05.595072 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.576117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.4.73.1:40319)
0810 06:25:05.580551 (+ 4434us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.580555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.580957 (+ 402us) server_negotiation.cc:407] Connection header received
0810 06:25:05.581004 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.581007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.581135 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.581219 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:05.582293 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.583149 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.584485 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.584967 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.585068 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.589703 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:05.589716 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:05.589734 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:05.589760 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:05.591945 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:05.592471 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:05.592479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:05.592480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:05.592646 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:05.592954 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:05.592958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:05.592959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:05.593264 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:05.593479 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.594202 (+ 723us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.594325 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4341,"thread_start_us":71,"threads_started":1}
I20260810 06:25:05.595671 6234 catalog_manager.cc:4841] TS ed05f1efc0ad4c8fb1f9b07f564252d3 (127.4.73.3:36325): tablet f62389f3a36d4ea181f7196802e25015 (table test-table [id=bbcba5a51fd84be8a7b59fdd7191fc24]) successfully deleted
W20260810 06:25:05.596487 6607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38656
I20260810 06:25:05.619154 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:05.600947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.4.73.62:44241)
0810 06:25:05.604516 (+ 3569us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:05.604521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:05.604974 (+ 453us) server_negotiation.cc:407] Connection header received
0810 06:25:05.605030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:05.605033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:05.605089 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:05.605184 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:05.606222 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.608216 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.609890 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:05.610454 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:05.610740 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:05.613034 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:05.613038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:05.613041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:05.613107 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:05.615593 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:05.616631 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:05.616635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:05.616637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:05.616823 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:05.617482 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:05.617486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:05.617488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:05.617947 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:05.618082 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:05.618780 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 06:25:05.618971 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3477,"thread_start_us":78,"threads_started":1}
W20260810 06:25:05.620146 6260 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35816
I20260810 06:25:05.623651 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6387
I20260810 06:25:05.642799 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6631
I20260810 06:25:05.657554 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6863
I20260810 06:25:05.672226 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6202
2026-08-10T06:25:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3647 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:05 dist-test-slave-45dx krb5kdc[7277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:05 dist-test-slave-45dx krb5kdc[7277](info): set up 2 sockets
Aug 10 06:25:05 dist-test-slave-45dx krb5kdc[7277](info): commencing operation
krb5kdc: starting...
W20260810 06:25:07.057427 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.349s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:07.173358 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:42473
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:42473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:07.284747 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:07.285018 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:07.285082 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:07.289110 7299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:07.289182 7299 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 06:25:07.289211 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:07.289251 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:07.289288 7299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:07.289325 7299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:07.294457 7299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:42473
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:42473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:07.295416 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:07.296422 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:07.304925 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:07.305030 7299 server_base.cc:1034] running on GCE node
W20260810 06:25:07.305032 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:07.304932 7314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:07.305838 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:07.306845 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:07.308027 7299 hybrid_clock.cc:648] HybridClock initialized: now 1786343107308009 us; error 43 us; skew 500 ppm
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:07.310600 7299 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:07.311476 7299 webserver.cc:460] Webserver started at http://127.4.73.62:32789/ using document root <none> and password file <none>
I20260810 06:25:07.312014 7299 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:07.312112 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:07.312384 7299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:07.314322 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "896dcb4190930db43f990d32d8fa62f5"
server_key_iv: "e5fd5029cb1b74fcdd43c18e8a75d7f8"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.314888 7299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "896dcb4190930db43f990d32d8fa62f5"
server_key_iv: "e5fd5029cb1b74fcdd43c18e8a75d7f8"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.318480 7299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:25:07.320778 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:07.321946 7299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:07.322090 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "896dcb4190930db43f990d32d8fa62f5"
server_key_iv: "e5fd5029cb1b74fcdd43c18e8a75d7f8"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.322196 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:07.336047 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:07.338699 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:07.338894 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:07.346675 7299 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:42473
I20260810 06:25:07.346701 7427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:42473 every 8 connection(s)
I20260810 06:25:07.347505 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:07.349139 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7299
I20260810 06:25:07.349232 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:07.349495 4388 external_mini_cluster.cc:1428] Setting key a347e16bbab9279e15b32718f2d048df
I20260810 06:25:07.350103 7428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:07.357092 7428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Bootstrap starting.
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:07.359354 7428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:07.360080 7428 log.cc:824] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:07.361892 7428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: No bootstrap required, opened a new log
I20260810 06:25:07.363910 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.350829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.4.73.62:42473)
0810 06:25:07.351257 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:07.351270 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:07.351303 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:25:07.351936 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:07.351950 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:07.352235 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:07.352554 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:07.353455 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.354382 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.355566 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.356003 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.356116 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.358728 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:07.358732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:07.358743 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:07.358767 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:07.361242 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.361667 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.361673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.361679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.361799 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.362083 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.362086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.362088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.362477 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:07.362685 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:07.362972 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 06:25:07.363258 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":154,"threads_started":1}
I20260810 06:25:07.365164 7428 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42473 } }
I20260810 06:25:07.365394 7428 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:07.365480 7428 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6773bcde6d4e3e8923fa6dffbe65fa, State: Initialized, Role: FOLLOWER
I20260810 06:25:07.365943 7428 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42473 } }
I20260810 06:25:07.366115 7428 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:07.366183 7428 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:07.366273 7428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:07.367026 7428 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42473 } }
I20260810 06:25:07.367297 7428 leader_election.cc:302] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd6773bcde6d4e3e8923fa6dffbe65fa; no voters:
I20260810 06:25:07.367542 7428 leader_election.cc:288] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:07.367669 7438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:07.367883 7438 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [term 1 LEADER]: Becoming Leader. State: Replica: cd6773bcde6d4e3e8923fa6dffbe65fa, State: Running, Role: LEADER
I20260810 06:25:07.368160 7438 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42473 } }
I20260810 06:25:07.368687 7428 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:07.370378 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42473 } } }
I20260810 06:25:07.370543 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:07.370854 7441 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd6773bcde6d4e3e8923fa6dffbe65fa. Latest consensus state: current_term: 1 leader_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd6773bcde6d4e3e8923fa6dffbe65fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42473 } } }
I20260810 06:25:07.370955 7441 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:07.371219 7464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:07.373726 7464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:07.378836 7464 catalog_manager.cc:1347] Generated new cluster ID: 081ed2ff0a704886adbb1a3eba0a0dfb
I20260810 06:25:07.378896 7464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:07.393774 7464 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:07.394603 7464 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:07.400830 7464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Generated new TSK 0
I20260810 06:25:07.401441 7464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:07.429145 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42473
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:07.537757 7479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:07.538007 7479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:07.538075 7479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:07.541913 7479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:07.541982 7479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:07.542044 7479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 06:25:07.542083 7479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:07.542148 7479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:07.547240 7479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:42473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:07.548122 7479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:07.549049 7479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:07.558171 7489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:07.558209 7488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:07.558230 7491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:07.558344 7479 server_base.cc:1034] running on GCE node
I20260810 06:25:07.558751 7479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:07.559405 7479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:07.560561 7479 hybrid_clock.cc:648] HybridClock initialized: now 1786343107560547 us; error 26 us; skew 500 ppm
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:07.563365 7479 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:07.564132 7479 webserver.cc:460] Webserver started at http://127.4.73.1:38995/ using document root <none> and password file <none>
I20260810 06:25:07.564733 7479 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:07.564795 7479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:07.564955 7479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:07.566883 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "02ac2c1a700b441aa7fd74096b860ccf"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "f37cfad5c09d211fce984861c511c50f"
server_key_iv: "ec3a79d5375bdb7585cede0168dd891c"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.567407 7479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "02ac2c1a700b441aa7fd74096b860ccf"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "f37cfad5c09d211fce984861c511c50f"
server_key_iv: "ec3a79d5375bdb7585cede0168dd891c"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.571169 7479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 06:25:07.573452 7503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:07.574604 7479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:07.574731 7479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "02ac2c1a700b441aa7fd74096b860ccf"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "f37cfad5c09d211fce984861c511c50f"
server_key_iv: "ec3a79d5375bdb7585cede0168dd891c"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.574855 7479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:07.588608 7479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:07.591979 7479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:07.592279 7479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:07.593930 7479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:07.594010 7479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:07.594086 7479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:07.594130 7479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:07.606011 7479 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:40871
I20260810 06:25:07.606074 7701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:40871 every 8 connection(s)
I20260810 06:25:07.606868 7479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:07.610461 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.594982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:55249 (local address 127.4.73.62:42473)
0810 06:25:07.595148 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:07.595153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:07.595930 (+ 777us) server_negotiation.cc:407] Connection header received
0810 06:25:07.597038 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:07.597042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:07.597099 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:07.597184 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:07.598737 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.599710 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.601657 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.602004 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.602133 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.606286 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:07.606289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:07.606291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:07.606316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:07.608428 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.609056 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.609059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.609060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.609142 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.609465 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.609468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.609469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.609650 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:07.609728 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:07.610224 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 06:25:07.610341 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 06:25:07.611253 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.595309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42473 (local address 127.4.73.1:55249)
0810 06:25:07.595750 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:07.595811 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:07.596808 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:07.597399 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:07.597410 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:07.597943 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:07.598530 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.598544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.599886 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.599890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.601511 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.601523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.602183 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.602187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.602386 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.603892 (+ 1506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:07.603915 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:07.606102 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:07.608548 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.608554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.608567 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.608934 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.609226 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.609228 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.609230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.609361 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.609773 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:07.609778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:07.610007 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 06:25:07.610159 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 06:25:07.610397 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":138,"threads_started":1}
I20260810 06:25:07.612648 7704 heartbeater.cc:347] Connected to a master server at 127.4.73.62:42473
I20260810 06:25:07.612790 7704 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:07.613343 7704 heartbeater.cc:511] Master 127.4.73.62:42473 requested a full tablet report, sending...
I20260810 06:25:07.615002 7353 ts_manager.cc:194] Registered new tserver with Master: 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871)
I20260810 06:25:07.615614 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7479
I20260810 06:25:07.615733 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:07.616017 4388 external_mini_cluster.cc:1428] Setting key d956d0ffeab70b35e4b2624bef3bef25
I20260810 06:25:07.616623 7353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:55249
I20260810 06:25:07.623030 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.615025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:54671 (local address 127.4.73.62:42473)
0810 06:25:07.615233 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:07.615237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:07.615348 (+ 111us) server_negotiation.cc:407] Connection header received
0810 06:25:07.615461 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:07.615463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:07.615513 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:07.615606 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:07.616167 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.616972 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.618164 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.618486 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.618603 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.619937 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:07.619941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:07.619943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:07.619967 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:07.621394 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.621847 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.621850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.621852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.621937 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.622221 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.622225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.622227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.622403 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:07.622512 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:07.622778 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 06:25:07.622891 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260810 06:25:07.623042 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.614953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42473 (local address 127.4.73.1:54671)
0810 06:25:07.615231 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:07.615247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:07.615365 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:07.615639 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:07.615642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:07.615868 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:07.615982 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.615989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.617107 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.617110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.618027 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.618041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.618623 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.618629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.618755 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.619244 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:07.619270 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:07.619791 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:07.621521 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.621525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.621527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.621728 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.622032 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.622036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.622038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.622128 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.622504 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:07.622507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:07.622604 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 06:25:07.622710 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 06:25:07.622847 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:07.643415 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42473
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:07.755997 7718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:07.756214 7718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:07.756253 7718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:07.760167 7718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:07.760219 7718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:07.760318 7718 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 06:25:07.760355 7718 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:07.760428 7718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:07.765472 7718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:42473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:07.766382 7718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:07.767302 7718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:07.777045 7728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:07.777078 7731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:07.777091 7724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:07.777493 7718 server_base.cc:1034] running on GCE node
I20260810 06:25:07.777920 7718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:07.778528 7718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:07.779723 7718 hybrid_clock.cc:648] HybridClock initialized: now 1786343107779710 us; error 25 us; skew 500 ppm
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:07.782523 7718 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:07.783392 7718 webserver.cc:460] Webserver started at http://127.4.73.2:44427/ using document root <none> and password file <none>
I20260810 06:25:07.783919 7718 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:07.784010 7718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:07.784233 7718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:07.786204 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "3b538cbd3e374000898a61d1c985bf1c"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "0dc7e613a70bf649ea27bd146590ba1e"
server_key_iv: "7903b080777682c722d1434fc05aca52"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.786760 7718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "3b538cbd3e374000898a61d1c985bf1c"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "0dc7e613a70bf649ea27bd146590ba1e"
server_key_iv: "7903b080777682c722d1434fc05aca52"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.790356 7718 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:25:07.792591 7746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:07.793875 7718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:07.794016 7718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "3b538cbd3e374000898a61d1c985bf1c"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "0dc7e613a70bf649ea27bd146590ba1e"
server_key_iv: "7903b080777682c722d1434fc05aca52"
server_key_version: "encryptionkey@0"
I20260810 06:25:07.794134 7718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:07.816247 7718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:07.818886 7718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:07.819085 7718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:07.820631 7718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:07.820708 7718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:07.820786 7718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:07.820842 7718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:07.832080 7718 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:41629
I20260810 06:25:07.832161 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:41629 every 8 connection(s)
I20260810 06:25:07.832974 7718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:07.835280 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.821374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:35555 (local address 127.4.73.62:42473)
0810 06:25:07.821508 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:07.821511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:07.823126 (+ 1615us) server_negotiation.cc:407] Connection header received
0810 06:25:07.824434 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:07.824438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:07.824497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:07.824590 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:07.826241 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.827137 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.828423 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.828660 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.828776 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.831434 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:07.831441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:07.831446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:07.831480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:07.833163 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.833761 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.833765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.833767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.833884 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.834254 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.834256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.834257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.834424 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:07.834499 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:07.835068 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 06:25:07.835164 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:25:07.836059 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.822457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42473 (local address 127.4.73.2:35555)
0810 06:25:07.822938 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:07.823015 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:07.824174 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:07.824810 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:07.824819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:07.825317 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:07.826048 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.826070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.827253 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.827257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.828247 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.828258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.828826 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.828830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.829003 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.829616 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:07.829632 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:07.831269 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:07.833274 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.833280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.833293 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.833635 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.834025 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.834027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.834029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.834164 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.834638 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:07.834643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:07.834857 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 06:25:07.835030 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 06:25:07.835257 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260810 06:25:07.837396 7941 heartbeater.cc:347] Connected to a master server at 127.4.73.62:42473
I20260810 06:25:07.837567 7941 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:07.838044 7941 heartbeater.cc:511] Master 127.4.73.62:42473 requested a full tablet report, sending...
I20260810 06:25:07.839030 7353 ts_manager.cc:194] Registered new tserver with Master: 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629)
I20260810 06:25:07.839704 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7718
I20260810 06:25:07.839772 7353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:35555
I20260810 06:25:07.839840 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:07.840135 4388 external_mini_cluster.cc:1428] Setting key 27edcc398d21dc63c00d973e4fba9034
I20260810 06:25:07.847600 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.839447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:46905 (local address 127.4.73.62:42473)
0810 06:25:07.840576 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:07.840579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:07.840590 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:25:07.840618 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:07.840620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:07.840651 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:07.840741 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:07.841231 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.841934 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.843174 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.843366 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.843469 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.844588 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:07.844591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:07.844594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:07.844616 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:07.846082 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.846490 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.846494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.846495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.846571 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:07.846839 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:07.846842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:07.846844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:07.847037 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:07.847137 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:07.847376 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 06:25:07.847473 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1063}
I20260810 06:25:07.847609 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:07.839388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42473 (local address 127.4.73.2:46905)
0810 06:25:07.839698 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:07.839713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:07.839814 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:07.840757 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:07.840760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:07.840969 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:07.841102 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.841109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.842074 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.842080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.843048 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:07.843063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:07.843475 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:07.843479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:07.843570 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:07.844068 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:07.844089 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:07.844442 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:07.846194 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.846197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.846199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.846385 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.846656 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:07.846659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:07.846661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:07.846745 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:07.847130 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:07.847133 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:07.847224 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 06:25:07.847323 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 06:25:07.847451 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:07.866827 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42473
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:07.976622 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:07.976845 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:07.976886 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:07.980867 7960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:07.980934 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:07.980974 7960 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 06:25:07.980997 7960 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:07.981045 7960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:07.985957 7960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:42473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:07.986809 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:07.987713 7960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:07.996973 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:07.997153 7960 server_base.cc:1034] running on GCE node
W20260810 06:25:07.997038 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:07.996996 7973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:07.997615 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:07.998234 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:07.999414 7960 hybrid_clock.cc:648] HybridClock initialized: now 1786343107999397 us; error 40 us; skew 500 ppm
Aug 10 06:25:07 dist-test-slave-45dx krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:08.002094 7960 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:08.002972 7960 webserver.cc:460] Webserver started at http://127.4.73.3:38229/ using document root <none> and password file <none>
I20260810 06:25:08.003454 7960 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:08.003510 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:08.003659 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:08.005682 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "0c8c215936494f9db6ed8f8288d61e83"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "2ba74363d668d1b956843fbccab95ca1"
server_key_iv: "604c59a9ae5d2e60df08f8ddb0aef68e"
server_key_version: "encryptionkey@0"
I20260810 06:25:08.006183 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "0c8c215936494f9db6ed8f8288d61e83"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "2ba74363d668d1b956843fbccab95ca1"
server_key_iv: "604c59a9ae5d2e60df08f8ddb0aef68e"
server_key_version: "encryptionkey@0"
I20260810 06:25:08.009670 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:25:08.011857 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:08.013073 7960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:08.013192 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "0c8c215936494f9db6ed8f8288d61e83"
format_stamp: "Formatted at 2026-08-10 06:25:07 on dist-test-slave-45dx"
server_key: "2ba74363d668d1b956843fbccab95ca1"
server_key_iv: "604c59a9ae5d2e60df08f8ddb0aef68e"
server_key_version: "encryptionkey@0"
I20260810 06:25:08.013288 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:08.028059 7960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:08.030656 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:08.030882 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:08.032272 7960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:08.032377 7960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:08.032451 7960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:08.032502 7960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:08 dist-test-slave-45dx krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:08.043579 7960 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:46175
I20260810 06:25:08.044497 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:08.044461 8172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:46175 every 8 connection(s)
I20260810 06:25:08.053154 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7960
I20260810 06:25:08.053292 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:08.053663 4388 external_mini_cluster.cc:1428] Setting key 018d6949fc42fb937cae1596e093768b
I20260810 06:25:08.055200 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.033874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42473 (local address 127.4.73.3:40525)
0810 06:25:08.034379 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.034447 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.035680 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.036331 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.036342 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.036863 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:08.037559 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.037580 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.038510 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.038514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.039583 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.039593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.040112 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.040123 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.040393 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.041004 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:08.041020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:08.042810 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:08.044811 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:08.044816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:08.044830 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:08.045188 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:08.045505 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:08.045508 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:08.045510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:08.045644 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:08.046083 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:08.046090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:08.046328 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.053766 (+ 7438us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.054131 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260810 06:25:08.056169 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.033071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:40525 (local address 127.4.73.62:42473)
0810 06:25:08.033275 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.033287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.034563 (+ 1276us) server_negotiation.cc:407] Connection header received
0810 06:25:08.035945 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.035949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.036017 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.036102 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:08.037721 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.038373 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.039709 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.039941 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.040053 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.042990 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:08.042994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:08.042997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:08.043029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:08.044677 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.045318 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.045321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.045323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.045414 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.045744 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.045747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.045749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.045958 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:08.046083 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.055866 (+ 9783us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.056025 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 06:25:08.057369 8173 heartbeater.cc:347] Connected to a master server at 127.4.73.62:42473
I20260810 06:25:08.057583 8173 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:08.058189 8173 heartbeater.cc:511] Master 127.4.73.62:42473 requested a full tablet report, sending...
I20260810 06:25:08.059329 7352 ts_manager.cc:194] Registered new tserver with Master: 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:08.060195 7352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:40525
I20260810 06:25:08.066748 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.059592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:35761 (local address 127.4.73.62:42473)
0810 06:25:08.059688 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.059693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.060064 (+ 371us) server_negotiation.cc:407] Connection header received
0810 06:25:08.060174 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.060178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.060215 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.060334 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:08.060792 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.061348 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.062230 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.062457 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.062583 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.063667 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:08.063670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:08.063672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:08.063694 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:08.065182 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.065611 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.065615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.065616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.065696 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.065981 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.065986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.065987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.066163 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:08.066267 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.066512 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.066614 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 06:25:08.066767 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.059710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42473 (local address 127.4.73.3:35761)
0810 06:25:08.059974 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.059987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.060101 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.060347 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.060349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.060537 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:08.060654 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.060662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.061480 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.061482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.062092 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.062099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.062604 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.062608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.062707 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.063224 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:08.063234 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:08.063529 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:08.065293 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:08.065296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:08.065298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:08.065503 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:08.065799 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:08.065802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:08.065804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:08.065888 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:08.066258 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:08.066261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:08.066348 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.066458 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.066582 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":24,"thread_start_us":123,"threads_started":1}
I20260810 06:25:08.069615 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:25:08 dist-test-slave-45dx krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:25:08 dist-test-slave-45dx krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:08.102324 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.092560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.4.73.62:42473)
0810 06:25:08.092675 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.092684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.092695 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:25:08.092730 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.092732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.092767 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.092833 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:08.093895 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.094482 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.095921 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.096132 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.096206 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.099211 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:08.099215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:08.099217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:08.099238 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:08.100658 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.101068 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.101072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.101074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.101155 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.101434 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.101438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.101440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.101604 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:08.101736 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.102000 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.102124 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:25:08.105574 7352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:08.111008 7352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:08.128341 8200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.121107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:46175 (local address 127.0.0.1:44970)
0810 06:25:08.122372 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.122386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.122762 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.123454 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.123457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.123471 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.123590 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.123599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.125040 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.125044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.126411 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.126422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.127708 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.127711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.127835 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.127881 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.128010 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.128147 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":25,"spinlock_wait_cycles":621568,"thread_start_us":86,"threads_started":1}
I20260810 06:25:08.129026 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.121341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.4.73.3:46175)
0810 06:25:08.121716 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.121730 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.122497 (+ 767us) server_negotiation.cc:407] Connection header received
0810 06:25:08.122947 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.122952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.123153 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.123311 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.123718 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.124892 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.126560 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.127360 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.127461 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.127645 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.128742 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.128850 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
I20260810 06:25:08.129418 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.120516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:41629 (local address 127.0.0.1:57034)
0810 06:25:08.122024 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.122041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.122762 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.123335 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.123341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.123363 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.123517 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.123525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.125971 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.125974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.127274 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.127289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.128984 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.128990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.129078 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.129100 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.129203 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.129278 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"mutex_wait_us":22,"spinlock_wait_cycles":1261056,"thread_start_us":84,"threads_started":1}
I20260810 06:25:08.129459 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.121179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.4.73.2:41629)
0810 06:25:08.121851 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.121866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.122173 (+ 307us) server_negotiation.cc:407] Connection header received
0810 06:25:08.122947 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.122952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.123153 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.123676 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.123818 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.125844 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.128010 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.128847 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.128960 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.129132 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.129226 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.129320 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":95,"threads_started":1}
I20260810 06:25:08.130169 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.120409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40871 (local address 127.0.0.1:46142)
0810 06:25:08.121565 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.121615 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.122714 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.123649 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.123655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.123682 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.123805 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.123814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.125512 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.125519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.127473 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.127482 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.129305 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.129308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.129400 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.129422 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.129863 (+ 441us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.129970 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":55,"thread_start_us":66,"threads_started":1}
I20260810 06:25:08.130995 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.120541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.4.73.1:40871)
0810 06:25:08.121104 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.121111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.122618 (+ 1507us) server_negotiation.cc:407] Connection header received
0810 06:25:08.122926 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.122935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.123164 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.123310 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.123960 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.125283 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.127599 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.128899 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.129030 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.130614 (+ 1584us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.130677 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.130811 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":95,"threads_started":1}
I20260810 06:25:08.132390 7812 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef73834377946769e08d8c8b16fe301 (DEFAULT_TABLE table=test-table [id=82224b83b709456aa64231188195b26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:08.133563 7812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef73834377946769e08d8c8b16fe301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.134902 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef73834377946769e08d8c8b16fe301 (DEFAULT_TABLE table=test-table [id=82224b83b709456aa64231188195b26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:08.132949 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef73834377946769e08d8c8b16fe301 (DEFAULT_TABLE table=test-table [id=82224b83b709456aa64231188195b26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:08.135905 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef73834377946769e08d8c8b16fe301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.136595 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef73834377946769e08d8c8b16fe301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.144505 8207 tablet_bootstrap.cc:492] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Bootstrap starting.
I20260810 06:25:08.144661 8213 tablet_bootstrap.cc:492] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Bootstrap starting.
I20260810 06:25:08.145468 8206 tablet_bootstrap.cc:492] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Bootstrap starting.
I20260810 06:25:08.146929 8213 tablet_bootstrap.cc:654] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.147655 8207 tablet_bootstrap.cc:654] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.147825 8213 log.cc:824] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:08.148115 8206 tablet_bootstrap.cc:654] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.148500 8207 log.cc:824] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:08.149142 8206 log.cc:824] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:08.149922 8213 tablet_bootstrap.cc:492] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: No bootstrap required, opened a new log
I20260810 06:25:08.150131 8213 ts_tablet_manager.cc:1397] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:25:08.150399 8207 tablet_bootstrap.cc:492] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: No bootstrap required, opened a new log
I20260810 06:25:08.150583 8207 ts_tablet_manager.cc:1397] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:25:08.151999 8206 tablet_bootstrap.cc:492] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: No bootstrap required, opened a new log
I20260810 06:25:08.152186 8206 ts_tablet_manager.cc:1397] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 06:25:08.154090 8213 raft_consensus.cc:348] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.154347 8213 raft_consensus.cc:374] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.154402 8213 raft_consensus.cc:729] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8c215936494f9db6ed8f8288d61e83, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.154868 8213 consensus_queue.cc:260] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.155107 8207 raft_consensus.cc:348] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.155333 8207 raft_consensus.cc:374] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.155395 8207 raft_consensus.cc:729] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b538cbd3e374000898a61d1c985bf1c, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.155848 8207 consensus_queue.cc:260] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.156263 8213 ts_tablet_manager.cc:1428] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:25:08.156643 8173 heartbeater.cc:503] Master 127.4.73.62:42473 was elected leader, sending a full tablet report...
I20260810 06:25:08.156966 8207 ts_tablet_manager.cc:1428] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260810 06:25:08.157697 7941 heartbeater.cc:503] Master 127.4.73.62:42473 was elected leader, sending a full tablet report...
I20260810 06:25:08.158751 8206 raft_consensus.cc:348] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.159094 8206 raft_consensus.cc:374] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.159209 8206 raft_consensus.cc:729] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ac2c1a700b441aa7fd74096b860ccf, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.159660 8206 consensus_queue.cc:260] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.160795 8206 ts_tablet_manager.cc:1428] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260810 06:25:08.161124 7704 heartbeater.cc:503] Master 127.4.73.62:42473 was elected leader, sending a full tablet report...
W20260810 06:25:08.295763 8174 tablet.cc:2369] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:08.321434 8218 raft_consensus.cc:482] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:08.321596 8218 raft_consensus.cc:504] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.322650 8218 leader_election.cc:288] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629)
I20260810 06:25:08.326705 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.323033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:43123 (local address 127.4.73.1:40871)
0810 06:25:08.323171 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.323175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.323191 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:25:08.323248 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.323252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.323301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.323397 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.323767 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.324404 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.325620 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.326119 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.326258 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.326345 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.326497 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.326583 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:25:08.326700 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.322901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40871 (local address 127.4.73.3:43123)
0810 06:25:08.323041 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.323059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.323157 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.323425 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.323428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.323448 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.323623 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.323631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.324513 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.324517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.325497 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.325507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.326246 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.326250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.326372 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.326394 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.326468 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.326545 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 06:25:08.326839 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.323021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:58383 (local address 127.4.73.2:41629)
0810 06:25:08.323215 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.323219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.323534 (+ 315us) server_negotiation.cc:407] Connection header received
0810 06:25:08.323624 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.323627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.323684 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.323767 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.324039 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.324639 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.325754 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.326265 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.326367 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.326429 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.326647 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.326717 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 06:25:08.327014 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.322901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:41629 (local address 127.4.73.3:58383)
0810 06:25:08.323415 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.323428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.323541 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.323783 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.323786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.323804 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.323921 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.323927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.324752 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.324755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.325631 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.325641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.326393 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.326396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.326501 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.326525 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.326598 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.326660 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":140,"thread_start_us":116,"threads_started":1}
I20260810 06:25:08.327452 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef73834377946769e08d8c8b16fe301" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf" is_pre_election: true
I20260810 06:25:08.327605 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef73834377946769e08d8c8b16fe301" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b538cbd3e374000898a61d1c985bf1c" is_pre_election: true
I20260810 06:25:08.327826 7627 raft_consensus.cc:2454] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8c215936494f9db6ed8f8288d61e83 in term 0.
I20260810 06:25:08.327929 7857 raft_consensus.cc:2454] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8c215936494f9db6ed8f8288d61e83 in term 0.
I20260810 06:25:08.328379 7989 leader_election.cc:302] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 0c8c215936494f9db6ed8f8288d61e83; no voters:
I20260810 06:25:08.328617 8218 raft_consensus.cc:2781] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:08.328739 8218 raft_consensus.cc:482] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:08.328795 8218 raft_consensus.cc:3037] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.329782 8218 raft_consensus.cc:504] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.330124 8218 leader_election.cc:288] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 election: Requested vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629)
I20260810 06:25:08.330438 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef73834377946769e08d8c8b16fe301" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf"
I20260810 06:25:08.330545 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef73834377946769e08d8c8b16fe301" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b538cbd3e374000898a61d1c985bf1c"
I20260810 06:25:08.330574 7627 raft_consensus.cc:3037] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.330673 7857 raft_consensus.cc:3037] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.331564 7627 raft_consensus.cc:2454] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8c215936494f9db6ed8f8288d61e83 in term 1.
I20260810 06:25:08.331750 7857 raft_consensus.cc:2454] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8c215936494f9db6ed8f8288d61e83 in term 1.
I20260810 06:25:08.331890 7989 leader_election.cc:302] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 0c8c215936494f9db6ed8f8288d61e83; no voters:
I20260810 06:25:08.332085 8218 raft_consensus.cc:2781] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:08.332269 8218 raft_consensus.cc:686] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8c215936494f9db6ed8f8288d61e83, State: Running, Role: LEADER
I20260810 06:25:08.332623 8218 consensus_queue.cc:237] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
W20260810 06:25:08.334115 7944 tablet.cc:2369] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:08.335650 7352 catalog_manager.cc:5495] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "0c8c215936494f9db6ed8f8288d61e83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } health_report { overall_health: HEALTHY } } }
W20260810 06:25:08.359331 7708 tablet.cc:2369] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 06:25:08 dist-test-slave-45dx krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343107, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:08.381729 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.373814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42473 (local address 127.0.0.1:33890)
0810 06:25:08.374087 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.374099 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.374218 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.374473 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.374476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.374879 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:08.374966 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.374972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.375704 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.375710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.376258 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.376266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.376748 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.376751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.376828 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.377342 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:08.377364 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:08.378892 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:08.380342 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:08.380354 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:08.380362 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:08.380609 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:08.380843 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:08.380845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:08.380847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:08.380940 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:08.381274 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:08.381279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:08.381380 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.381467 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.381588 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260810 06:25:08.381767 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.373857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.4.73.62:42473)
0810 06:25:08.373999 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.374006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.374176 (+ 170us) server_negotiation.cc:407] Connection header received
0810 06:25:08.374319 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.374322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.374368 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.374451 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:08.375089 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.375597 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.376437 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.376642 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.376748 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.379015 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:08.379018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:08.379020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:08.379070 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:08.380209 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.380692 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.380695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.380696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.380766 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:08.381027 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:08.381029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:08.381029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:08.381184 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:08.381256 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.381525 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.381642 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":46}
I20260810 06:25:08.385226 7352 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:33890:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 06:25:08.385845 7352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:08.390297 7591 tablet_service.cc:1467] Processing CreateTablet for tablet f032a15939ce4694a40b93439c069e08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0f794c563be48cd9073e2bb44b39dde]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:08.390401 7812 tablet_service.cc:1467] Processing CreateTablet for tablet f032a15939ce4694a40b93439c069e08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0f794c563be48cd9073e2bb44b39dde]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:08.391036 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f032a15939ce4694a40b93439c069e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.391088 7812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f032a15939ce4694a40b93439c069e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.393803 8207 tablet_bootstrap.cc:492] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: Bootstrap starting.
I20260810 06:25:08.393822 8206 tablet_bootstrap.cc:492] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: Bootstrap starting.
I20260810 06:25:08.394241 8063 tablet_service.cc:1467] Processing CreateTablet for tablet f032a15939ce4694a40b93439c069e08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0f794c563be48cd9073e2bb44b39dde]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:08.394753 8207 tablet_bootstrap.cc:654] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.394933 8206 tablet_bootstrap.cc:654] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.394941 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f032a15939ce4694a40b93439c069e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.396242 8206 tablet_bootstrap.cc:492] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: No bootstrap required, opened a new log
I20260810 06:25:08.396361 8206 ts_tablet_manager.cc:1397] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:25:08.396811 8206 raft_consensus.cc:348] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.396950 8206 raft_consensus.cc:374] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.397001 8206 raft_consensus.cc:729] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ac2c1a700b441aa7fd74096b860ccf, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.397135 8206 consensus_queue.cc:260] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.397405 8222 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.397554 8222 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:08.397878 8206 ts_tablet_manager.cc:1428] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:08.398149 8213 tablet_bootstrap.cc:492] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: Bootstrap starting.
I20260810 06:25:08.398926 8207 tablet_bootstrap.cc:492] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: No bootstrap required, opened a new log
I20260810 06:25:08.399027 8207 ts_tablet_manager.cc:1397] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260810 06:25:08.399232 8213 tablet_bootstrap.cc:654] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.399510 8207 raft_consensus.cc:348] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.399639 8207 raft_consensus.cc:374] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.399681 8207 raft_consensus.cc:729] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b538cbd3e374000898a61d1c985bf1c, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.399860 8207 consensus_queue.cc:260] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.400162 8207 ts_tablet_manager.cc:1428] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:25:08.400741 8220 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.400889 8220 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:08.400905 8213 tablet_bootstrap.cc:492] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: No bootstrap required, opened a new log
I20260810 06:25:08.400987 8213 ts_tablet_manager.cc:1397] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:25:08.401500 8213 raft_consensus.cc:348] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.401616 8213 raft_consensus.cc:374] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.401659 8213 raft_consensus.cc:729] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8c215936494f9db6ed8f8288d61e83, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.401855 8213 consensus_queue.cc:260] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.402136 8213 ts_tablet_manager.cc:1428] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:08.402100 8218 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.402216 8218 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:08.515417 8220 raft_consensus.cc:482] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:08.515559 8220 raft_consensus.cc:504] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.516825 8220 leader_election.cc:288] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:08.520819 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.517071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40871 (local address 127.4.73.2:56291)
0810 06:25:08.517386 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.517401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.517579 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.517930 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.517933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.517950 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.518079 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.518084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.518831 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.518835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.519636 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.519645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.520359 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.520363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.520478 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.520504 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.520596 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.520667 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":112,"threads_started":1}
I20260810 06:25:08.520803 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.517253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:56291 (local address 127.4.73.1:40871)
0810 06:25:08.517388 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.517394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.517491 (+ 97us) server_negotiation.cc:407] Connection header received
0810 06:25:08.517786 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.517789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.517828 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.517906 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.518189 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.518718 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.519764 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.520180 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.520295 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.520396 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.520615 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.520694 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:25:08.521018 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.517597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:50817 (local address 127.4.73.3:46175)
0810 06:25:08.517744 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.517748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.517989 (+ 241us) server_negotiation.cc:407] Connection header received
0810 06:25:08.518116 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.518119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.518165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.518248 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.518517 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.519029 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.520025 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.520479 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.520594 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.520651 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.520835 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.520915 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 06:25:08.521032 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.517505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:46175 (local address 127.4.73.2:50817)
0810 06:25:08.517901 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.517912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.518027 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.518262 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.518265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.518280 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.518404 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.518410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.519146 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.519150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.519897 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.519905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.520600 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.520603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.520700 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.520721 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.520818 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.520887 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":136,"threads_started":1}
I20260810 06:25:08.521251 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f032a15939ce4694a40b93439c069e08" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf" is_pre_election: true
I20260810 06:25:08.521407 7627 raft_consensus.cc:2454] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 0.
I20260810 06:25:08.521647 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f032a15939ce4694a40b93439c069e08" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83" is_pre_election: true
I20260810 06:25:08.521811 7754 leader_election.cc:302] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 3b538cbd3e374000898a61d1c985bf1c; no voters:
I20260810 06:25:08.521883 8092 raft_consensus.cc:2454] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 0.
I20260810 06:25:08.522104 8220 raft_consensus.cc:2781] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:08.522178 8220 raft_consensus.cc:482] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:08.522249 8220 raft_consensus.cc:3037] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.523341 8220 raft_consensus.cc:504] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.523671 8220 leader_election.cc:288] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 election: Requested vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:08.523996 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f032a15939ce4694a40b93439c069e08" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf"
I20260810 06:25:08.524114 7627 raft_consensus.cc:3037] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.524048 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f032a15939ce4694a40b93439c069e08" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83"
I20260810 06:25:08.524277 8092 raft_consensus.cc:3037] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.525172 7627 raft_consensus.cc:2454] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 1.
I20260810 06:25:08.525400 8092 raft_consensus.cc:2454] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 1.
I20260810 06:25:08.525440 7754 leader_election.cc:302] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 3b538cbd3e374000898a61d1c985bf1c; no voters:
I20260810 06:25:08.525599 8220 raft_consensus.cc:2781] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:08.525771 8220 raft_consensus.cc:686] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [term 1 LEADER]: Becoming Leader. State: Replica: 3b538cbd3e374000898a61d1c985bf1c, State: Running, Role: LEADER
I20260810 06:25:08.526149 8220 consensus_queue.cc:237] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.528331 8243 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: TxnStatusTablet state changed. Reason: New leader 3b538cbd3e374000898a61d1c985bf1c. Latest consensus state: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.528455 8243 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:08.528750 8245 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 06:25:08.528980 8245 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 06:25:08.529462 7353 catalog_manager.cc:5495] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2). New cstate: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } health_report { overall_health: UNKNOWN } } }
I20260810 06:25:08.548863 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.544898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.4.73.2:41629)
0810 06:25:08.545013 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.545016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.545029 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:08.545215 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.545217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.545267 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.545347 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:08.545611 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.546109 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.547369 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.547603 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.547691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.548003 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.548469 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.548557 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.548684 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.548734 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 06:25:08.548895 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.544753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:41629 (local address 127.0.0.1:57042)
0810 06:25:08.544952 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.544970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.545077 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.545382 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.545385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.545396 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:08.545505 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.545511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.546206 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.546209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.547220 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.547230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.547717 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.547721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.547815 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.547867 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.548558 (+ 691us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.548576 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.548665 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.548756 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 06:25:08.553869 7627 raft_consensus.cc:1261] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Refusing update from remote peer 3b538cbd3e374000898a61d1c985bf1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:08.553869 8092 raft_consensus.cc:1261] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Refusing update from remote peer 3b538cbd3e374000898a61d1c985bf1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:08.554499 8243 consensus_queue.cc:1035] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:08.554680 8220 consensus_queue.cc:1035] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:08.558233 8218 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: TxnStatusTablet state changed. Reason: New leader 3b538cbd3e374000898a61d1c985bf1c. Latest consensus state: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.558367 8218 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:08.560343 8222 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: TxnStatusTablet state changed. Reason: New leader 3b538cbd3e374000898a61d1c985bf1c. Latest consensus state: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.560474 8222 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:08.563954 8243 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.564103 8243 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:08.564483 8218 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.564577 8218 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 0c8c215936494f9db6ed8f8288d61e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:08.564587 8220 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.564684 8220 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 3b538cbd3e374000898a61d1c985bf1c: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:08.564965 8249 mvcc.cc:205] Tried to move back new op lower bound from 7316861372629602304 to 7316861372525486080. Current Snapshot: MvccSnapshot[applied={T|T < 7316861372629602304}]
I20260810 06:25:08.565155 8250 mvcc.cc:205] Tried to move back new op lower bound from 7316861372629602304 to 7316861372525486080. Current Snapshot: MvccSnapshot[applied={T|T < 7316861372629602304}]
I20260810 06:25:08.566897 8222 tablet_replica.cc:439] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } } }
I20260810 06:25:08.567008 8222 tablet_replica.cc:442] T f032a15939ce4694a40b93439c069e08 P 02ac2c1a700b441aa7fd74096b860ccf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:08.576723 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.573215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.4.73.3:46175)
0810 06:25:08.573312 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.573316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.573376 (+ 60us) server_negotiation.cc:407] Connection header received
0810 06:25:08.573557 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.573560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.573612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.573708 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:08.573963 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.574462 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.575352 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.575576 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.575684 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.575975 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.576370 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.576483 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.576544 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.576594 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:25:08.587036 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.583503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:51061 (local address 127.4.73.2:41629)
0810 06:25:08.583618 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.583622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.583756 (+ 134us) server_negotiation.cc:407] Connection header received
0810 06:25:08.583878 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.583881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.583919 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.583977 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:08.584215 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.584803 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.586049 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.586257 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.586333 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.586686 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.586743 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.586796 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.586906 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.586946 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:25:08.587114 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.583356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:41629 (local address 127.4.73.3:51061)
0810 06:25:08.583673 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.583687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.583792 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.584009 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.584012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.584019 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:08.584119 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.584125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.584935 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.584939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.585928 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.585939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.586382 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.586386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.586496 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.586542 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.586818 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.586824 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.586890 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.586972 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":129,"threads_started":1}
I20260810 06:25:08.593703 7627 raft_consensus.cc:1261] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Refusing update from remote peer 0c8c215936494f9db6ed8f8288d61e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:08.593878 7857 raft_consensus.cc:1261] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Refusing update from remote peer 0c8c215936494f9db6ed8f8288d61e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:08.594131 8225 consensus_queue.cc:1035] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:08.594331 8218 consensus_queue.cc:1035] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:08.597869 8252 mvcc.cc:205] Tried to move back new op lower bound from 7316861372795920384 to 7316861371732754432. Current Snapshot: MvccSnapshot[applied={T|T < 7316861372795920384}]
I20260810 06:25:08.630249 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.626579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:43263 (local address 127.4.73.3:46175)
0810 06:25:08.626698 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.626703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.627010 (+ 307us) server_negotiation.cc:407] Connection header received
0810 06:25:08.627192 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.627195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.627241 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.627332 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:08.627602 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.628276 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.629222 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.629445 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.629550 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.629838 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.629913 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.629989 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.630078 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.630123 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:25:08.630285 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.626546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:46175 (local address 127.4.73.2:43263)
0810 06:25:08.626934 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.626950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.627088 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.627344 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.627346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.627354 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:08.627455 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.627462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.628467 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.628469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.629095 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.629103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.629555 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.629558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.629641 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.629690 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.629999 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.630007 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.630069 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.630151 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260810 06:25:08.658850 7352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:08.659062 7352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:08.661916 7352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Sending DeleteTablet for 3 replicas of tablet 7ef73834377946769e08d8c8b16fe301
I20260810 06:25:08.662693 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef73834377946769e08d8c8b16fe301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:08 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:44970
I20260810 06:25:08.663457 8288 tablet_replica.cc:330] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: stopping tablet replica
I20260810 06:25:08.663421 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef73834377946769e08d8c8b16fe301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:08 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260810 06:25:08.663865 8288 raft_consensus.cc:2229] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:08.664017 7812 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef73834377946769e08d8c8b16fe301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:08 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:57034
I20260810 06:25:08.664428 8288 raft_consensus.cc:2258] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:08.664515 7352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:08.664933 7352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:08.666092 8288 ts_tablet_manager.cc:1905] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:08.669479 8288 ts_tablet_manager.cc:1918] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:08.669690 8288 log.cc:1196] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/7ef73834377946769e08d8c8b16fe301
I20260810 06:25:08.670010 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 976a8c7479a7462ca14ec9df8eff5cda (DEFAULT_TABLE table=test-table [id=c037185a416d4f94abc8b4d0b68e92b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:08.670285 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976a8c7479a7462ca14ec9df8eff5cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.670293 8288 ts_tablet_manager.cc:1939] T 7ef73834377946769e08d8c8b16fe301 P 0c8c215936494f9db6ed8f8288d61e83: Deleting consensus metadata
I20260810 06:25:08.671660 7332 catalog_manager.cc:4841] TS 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175): tablet 7ef73834377946769e08d8c8b16fe301 (table test-table [id=82224b83b709456aa64231188195b26e]) successfully deleted
I20260810 06:25:08.672457 8213 tablet_bootstrap.cc:492] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: Bootstrap starting.
I20260810 06:25:08.672618 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 976a8c7479a7462ca14ec9df8eff5cda (DEFAULT_TABLE table=test-table [id=c037185a416d4f94abc8b4d0b68e92b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:08.672919 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976a8c7479a7462ca14ec9df8eff5cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.673244 8213 tablet_bootstrap.cc:654] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.674319 8213 tablet_bootstrap.cc:492] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: No bootstrap required, opened a new log
I20260810 06:25:08.674393 8213 ts_tablet_manager.cc:1397] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:08.674814 8213 raft_consensus.cc:348] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.674919 8213 raft_consensus.cc:374] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.674952 8213 raft_consensus.cc:729] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8c215936494f9db6ed8f8288d61e83, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.675078 8213 consensus_queue.cc:260] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.675333 8213 ts_tablet_manager.cc:1428] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:08.675623 8206 tablet_bootstrap.cc:492] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: Bootstrap starting.
I20260810 06:25:08.676857 8293 tablet_replica.cc:330] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: stopping tablet replica
I20260810 06:25:08.677089 8291 tablet_replica.cc:330] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: stopping tablet replica
I20260810 06:25:08.677460 8291 raft_consensus.cc:2229] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:08.676987 7812 tablet_service.cc:1467] Processing CreateTablet for tablet 976a8c7479a7462ca14ec9df8eff5cda (DEFAULT_TABLE table=test-table [id=c037185a416d4f94abc8b4d0b68e92b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:08.677804 7812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 976a8c7479a7462ca14ec9df8eff5cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:08.680207 8207 tablet_bootstrap.cc:492] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: Bootstrap starting.
I20260810 06:25:08.681144 8207 tablet_bootstrap.cc:654] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.681609 8293 raft_consensus.cc:2229] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:08.679643 8206 tablet_bootstrap.cc:654] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:08.682480 8291 raft_consensus.cc:2258] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:08.682451 8207 tablet_bootstrap.cc:492] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: No bootstrap required, opened a new log
I20260810 06:25:08.682657 8207 ts_tablet_manager.cc:1397] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 06:25:08.682700 8293 raft_consensus.cc:2258] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:08.683099 8207 raft_consensus.cc:348] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.683216 8207 raft_consensus.cc:374] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.683249 8207 raft_consensus.cc:729] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b538cbd3e374000898a61d1c985bf1c, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.683370 8207 consensus_queue.cc:260] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.683610 8207 ts_tablet_manager.cc:1428] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:08.684938 8293 ts_tablet_manager.cc:1905] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:08.687857 8293 ts_tablet_manager.cc:1918] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:08.687958 8293 log.cc:1196] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/7ef73834377946769e08d8c8b16fe301
I20260810 06:25:08.688274 8293 ts_tablet_manager.cc:1939] T 7ef73834377946769e08d8c8b16fe301 P 3b538cbd3e374000898a61d1c985bf1c: Deleting consensus metadata
I20260810 06:25:08.689536 7326 catalog_manager.cc:4841] TS 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629): tablet 7ef73834377946769e08d8c8b16fe301 (table test-table [id=82224b83b709456aa64231188195b26e]) successfully deleted
I20260810 06:25:08.690400 8206 tablet_bootstrap.cc:492] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: No bootstrap required, opened a new log
I20260810 06:25:08.690490 8206 ts_tablet_manager.cc:1397] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260810 06:25:08.690940 8206 raft_consensus.cc:348] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.691074 8206 raft_consensus.cc:374] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:08.691109 8206 raft_consensus.cc:729] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ac2c1a700b441aa7fd74096b860ccf, State: Initialized, Role: FOLLOWER
I20260810 06:25:08.691226 8206 consensus_queue.cc:260] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.691488 8206 ts_tablet_manager.cc:1428] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:25:08.693298 8291 ts_tablet_manager.cc:1905] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:08.695623 8291 ts_tablet_manager.cc:1918] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:08.695732 8291 log.cc:1196] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/7ef73834377946769e08d8c8b16fe301
I20260810 06:25:08.696013 8291 ts_tablet_manager.cc:1939] T 7ef73834377946769e08d8c8b16fe301 P 02ac2c1a700b441aa7fd74096b860ccf: Deleting consensus metadata
I20260810 06:25:08.697166 7332 catalog_manager.cc:4841] TS 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871): tablet 7ef73834377946769e08d8c8b16fe301 (table test-table [id=82224b83b709456aa64231188195b26e]) successfully deleted
I20260810 06:25:08.709578 8222 raft_consensus.cc:482] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:08.709683 8222 raft_consensus.cc:504] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.710871 8222 leader_election.cc:288] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:08.716380 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.711326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:35879 (local address 127.4.73.2:41629)
0810 06:25:08.711444 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.711448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.711714 (+ 266us) server_negotiation.cc:407] Connection header received
0810 06:25:08.711921 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.711928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.711975 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.712021 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.712528 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.713312 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.715257 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.715645 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.715761 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.715847 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.716141 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.716224 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:08.716495 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.711449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:48569 (local address 127.4.73.3:46175)
0810 06:25:08.711563 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.711568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.711790 (+ 222us) server_negotiation.cc:407] Connection header received
0810 06:25:08.711961 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.711965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.712017 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.712068 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:08.712528 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.713305 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.715260 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.715904 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.716018 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.716080 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.716285 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.716377 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:25:08.716521 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.711123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:46175 (local address 127.4.73.1:48569)
0810 06:25:08.711711 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.711724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.711849 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.712212 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.712214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.712234 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.712402 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.712407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.713484 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.713490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.715107 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.715118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.716034 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.716040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.716158 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.716178 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.716243 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.716356 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":137,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260810 06:25:08.717057 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.711011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:41629 (local address 127.4.73.1:35879)
0810 06:25:08.711634 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.711648 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.711793 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.712164 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.712170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.712221 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:08.712397 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.712403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.713464 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.713469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.715108 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.715118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.715853 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.715861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.715956 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.716005 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.716827 (+ 822us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.716907 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":188,"spinlock_wait_cycles":3840,"thread_start_us":68,"threads_started":1}
I20260810 06:25:08.717072 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a8c7479a7462ca14ec9df8eff5cda" candidate_uuid: "02ac2c1a700b441aa7fd74096b860ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83" is_pre_election: true
I20260810 06:25:08.717229 8092 raft_consensus.cc:2454] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ac2c1a700b441aa7fd74096b860ccf in term 0.
I20260810 06:25:08.717585 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a8c7479a7462ca14ec9df8eff5cda" candidate_uuid: "02ac2c1a700b441aa7fd74096b860ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b538cbd3e374000898a61d1c985bf1c" is_pre_election: true
I20260810 06:25:08.717738 7857 raft_consensus.cc:2454] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02ac2c1a700b441aa7fd74096b860ccf in term 0.
I20260810 06:25:08.718255 7513 leader_election.cc:302] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 0c8c215936494f9db6ed8f8288d61e83; no voters:
I20260810 06:25:08.718601 8222 raft_consensus.cc:2781] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:08.718675 8222 raft_consensus.cc:482] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:08.718714 8222 raft_consensus.cc:3037] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.719885 8222 raft_consensus.cc:504] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.720289 8222 leader_election.cc:288] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [CANDIDATE]: Term 1 election: Requested vote from peers 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:08.720589 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a8c7479a7462ca14ec9df8eff5cda" candidate_uuid: "02ac2c1a700b441aa7fd74096b860ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b538cbd3e374000898a61d1c985bf1c"
I20260810 06:25:08.720695 7857 raft_consensus.cc:3037] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.720736 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "976a8c7479a7462ca14ec9df8eff5cda" candidate_uuid: "02ac2c1a700b441aa7fd74096b860ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83"
I20260810 06:25:08.720850 8092 raft_consensus.cc:3037] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:08.721537 7857 raft_consensus.cc:2454] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ac2c1a700b441aa7fd74096b860ccf in term 1.
I20260810 06:25:08.721871 7504 leader_election.cc:302] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 3b538cbd3e374000898a61d1c985bf1c; no voters:
I20260810 06:25:08.721964 8092 raft_consensus.cc:2454] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02ac2c1a700b441aa7fd74096b860ccf in term 1.
I20260810 06:25:08.722039 8222 raft_consensus.cc:2781] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:08.722266 8222 raft_consensus.cc:686] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 1 LEADER]: Becoming Leader. State: Replica: 02ac2c1a700b441aa7fd74096b860ccf, State: Running, Role: LEADER
I20260810 06:25:08.722492 8222 consensus_queue.cc:237] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:08.724722 7353 catalog_manager.cc:5495] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1). New cstate: current_term: 1 leader_uuid: "02ac2c1a700b441aa7fd74096b860ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } health_report { overall_health: UNKNOWN } } }
I20260810 06:25:08.744524 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.740843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.4.73.1:40871)
0810 06:25:08.740948 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.740951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.740979 (+ 28us) server_negotiation.cc:407] Connection header received
0810 06:25:08.741141 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.741143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.741187 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.741266 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:08.741499 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.741991 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.743109 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.743332 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.743404 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.743676 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.744079 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.744159 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.744266 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.744382 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:25:08.753993 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.750998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:46461 (local address 127.4.73.2:41629)
0810 06:25:08.751178 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.751183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.751221 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:25:08.751352 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.751355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.751400 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.751470 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:08.751723 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.752168 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.752970 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.753199 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.753312 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.753583 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.753651 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.753727 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.753821 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.753866 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260810 06:25:08.754011 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.750909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:41629 (local address 127.4.73.1:46461)
0810 06:25:08.751158 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.751171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.751265 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.751489 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.751491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.751499 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:08.751628 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.751632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.752267 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.752272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.752857 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.752865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.753309 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.753312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.753403 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.753453 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.753736 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.753744 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.753813 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.753884 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":97,"threads_started":1}
I20260810 06:25:08.759963 8092 raft_consensus.cc:1261] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Refusing update from remote peer 02ac2c1a700b441aa7fd74096b860ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:08.760079 7857 raft_consensus.cc:1261] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Refusing update from remote peer 02ac2c1a700b441aa7fd74096b860ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:08.760500 8222 consensus_queue.cc:1035] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:08.760651 8302 consensus_queue.cc:1035] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:08.779060 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.775827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40871 (local address 127.4.73.2:55383)
0810 06:25:08.775945 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:08.775962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:08.776039 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:08.776324 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:08.776327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:08.776334 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:08.776425 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.776433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.777124 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.777129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.777752 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:08.777764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.778306 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.778309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:08.778425 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.778458 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.778743 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.778750 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:25:08.778824 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 06:25:08.778915 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 06:25:08.779599 8202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:08.775942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:55383 (local address 127.4.73.1:40871)
0810 06:25:08.776039 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:08.776043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:08.776056 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:08.776133 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:08.776136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:08.776178 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:08.776276 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:08.776535 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.777018 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.777888 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:08.778181 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:08.778298 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:08.778566 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:08.778648 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:08.778723 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:08.779443 (+ 720us) server_negotiation.cc:299] Negotiation successful
0810 06:25:08.779490 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 06:25:08.780169 8252 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 06:25:10.339391 7353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:10.339541 7353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:10.341790 7353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Sending DeleteTablet for 3 replicas of tablet 976a8c7479a7462ca14ec9df8eff5cda
I20260810 06:25:10.342298 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 976a8c7479a7462ca14ec9df8eff5cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:10 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260810 06:25:10.342470 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 976a8c7479a7462ca14ec9df8eff5cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:10 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:44970
I20260810 06:25:10.342568 7812 tablet_service.cc:1514] Processing DeleteTablet for tablet 976a8c7479a7462ca14ec9df8eff5cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:10 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:57034
I20260810 06:25:10.342852 8356 tablet_replica.cc:330] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: stopping tablet replica
I20260810 06:25:10.342886 8357 tablet_replica.cc:330] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: stopping tablet replica
I20260810 06:25:10.343001 8357 raft_consensus.cc:2229] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:10.343068 8356 raft_consensus.cc:2229] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:10.343158 8357 raft_consensus.cc:2258] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:10.343185 8356 raft_consensus.cc:2258] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:10.343776 8356 ts_tablet_manager.cc:1905] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:10.345110 8357 ts_tablet_manager.cc:1905] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:10.346663 8356 ts_tablet_manager.cc:1918] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:25:10.346812 8356 log.cc:1196] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/976a8c7479a7462ca14ec9df8eff5cda
I20260810 06:25:10.347254 8356 ts_tablet_manager.cc:1939] T 976a8c7479a7462ca14ec9df8eff5cda P 0c8c215936494f9db6ed8f8288d61e83: Deleting consensus metadata
I20260810 06:25:10.347836 8355 tablet_replica.cc:330] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: stopping tablet replica
I20260810 06:25:10.348021 8355 raft_consensus.cc:2229] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:10.348054 8357 ts_tablet_manager.cc:1918] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:25:10.348035 7332 catalog_manager.cc:4841] TS 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175): tablet 976a8c7479a7462ca14ec9df8eff5cda (table test-table [id=c037185a416d4f94abc8b4d0b68e92b7]) successfully deleted
I20260810 06:25:10.348182 8357 log.cc:1196] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/976a8c7479a7462ca14ec9df8eff5cda
I20260810 06:25:10.348441 8355 raft_consensus.cc:2258] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:10.348531 8357 ts_tablet_manager.cc:1939] T 976a8c7479a7462ca14ec9df8eff5cda P 3b538cbd3e374000898a61d1c985bf1c: Deleting consensus metadata
I20260810 06:25:10.349081 8355 ts_tablet_manager.cc:1905] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:10.349257 7326 catalog_manager.cc:4841] TS 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629): tablet 976a8c7479a7462ca14ec9df8eff5cda (table test-table [id=c037185a416d4f94abc8b4d0b68e92b7]) successfully deleted
I20260810 06:25:10.351760 8355 ts_tablet_manager.cc:1918] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:25:10.351841 8355 log.cc:1196] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/976a8c7479a7462ca14ec9df8eff5cda
I20260810 06:25:10.352115 8355 ts_tablet_manager.cc:1939] T 976a8c7479a7462ca14ec9df8eff5cda P 02ac2c1a700b441aa7fd74096b860ccf: Deleting consensus metadata
I20260810 06:25:10.352819 7332 catalog_manager.cc:4841] TS 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871): tablet 976a8c7479a7462ca14ec9df8eff5cda (table test-table [id=c037185a416d4f94abc8b4d0b68e92b7]) successfully deleted
Aug 10 06:25:10 dist-test-slave-45dx krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:25:10 dist-test-slave-45dx krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343110, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:10.373438 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:10.364730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.4.73.62:42473)
0810 06:25:10.364996 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:10.365025 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:10.365118 (+ 93us) server_negotiation.cc:407] Connection header received
0810 06:25:10.365281 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:10.365284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:10.365338 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:10.365437 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:10.366139 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:10.366696 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:10.367536 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:10.367776 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:10.367901 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:10.370266 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:10.370270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:10.370272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:10.370307 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:10.371636 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:10.372077 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:10.372081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:10.372083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:10.372173 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:10.372542 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:10.372545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:10.372547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:10.372738 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:10.372892 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:10.373138 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 06:25:10.373264 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260810 06:25:10.375880 7353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:10.376271 7353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:10.381023 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac (DEFAULT_TABLE table=test-table [id=64ba3a4398bc485fbd859efe69f5d436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:10.381282 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac (DEFAULT_TABLE table=test-table [id=64ba3a4398bc485fbd859efe69f5d436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:10.381356 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:10.381594 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:10.383432 8370 tablet_bootstrap.cc:492] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: Bootstrap starting.
I20260810 06:25:10.383632 7812 tablet_service.cc:1467] Processing CreateTablet for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac (DEFAULT_TABLE table=test-table [id=64ba3a4398bc485fbd859efe69f5d436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:10.383898 7812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:10.384259 8371 tablet_bootstrap.cc:492] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: Bootstrap starting.
I20260810 06:25:10.384262 8370 tablet_bootstrap.cc:654] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:10.385152 8371 tablet_bootstrap.cc:654] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:10.385628 8370 tablet_bootstrap.cc:492] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: No bootstrap required, opened a new log
I20260810 06:25:10.385728 8370 ts_tablet_manager.cc:1397] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 06:25:10.386139 8370 raft_consensus.cc:348] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.386283 8370 raft_consensus.cc:374] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:10.386328 8370 raft_consensus.cc:729] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ac2c1a700b441aa7fd74096b860ccf, State: Initialized, Role: FOLLOWER
I20260810 06:25:10.386474 8370 consensus_queue.cc:260] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.386780 8370 ts_tablet_manager.cc:1428] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:25:10.386945 8374 tablet_bootstrap.cc:492] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: Bootstrap starting.
I20260810 06:25:10.387758 8374 tablet_bootstrap.cc:654] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:10.387337 8371 tablet_bootstrap.cc:492] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: No bootstrap required, opened a new log
I20260810 06:25:10.388096 8371 ts_tablet_manager.cc:1397] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 06:25:10.388583 8371 raft_consensus.cc:348] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.388727 8371 raft_consensus.cc:374] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:10.388763 8371 raft_consensus.cc:729] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8c215936494f9db6ed8f8288d61e83, State: Initialized, Role: FOLLOWER
I20260810 06:25:10.389056 8374 tablet_bootstrap.cc:492] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: No bootstrap required, opened a new log
I20260810 06:25:10.389169 8374 ts_tablet_manager.cc:1397] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:10.389475 8371 consensus_queue.cc:260] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.389647 8374 raft_consensus.cc:348] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.389781 8374 raft_consensus.cc:374] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:10.389823 8374 raft_consensus.cc:729] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b538cbd3e374000898a61d1c985bf1c, State: Initialized, Role: FOLLOWER
I20260810 06:25:10.389925 8371 ts_tablet_manager.cc:1428] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:10.389971 8374 consensus_queue.cc:260] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.390252 8374 ts_tablet_manager.cc:1428] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:25:10.784647 8377 raft_consensus.cc:482] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:10.784647 8317 raft_consensus.cc:482] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:10.784832 8377 raft_consensus.cc:504] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.784825 8317 raft_consensus.cc:504] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.785300 8317 leader_election.cc:288] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:10.785315 8377 leader_election.cc:288] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629)
I20260810 06:25:10.785825 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b538cbd3e374000898a61d1c985bf1c" is_pre_election: true
I20260810 06:25:10.785825 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83" is_pre_election: true
I20260810 06:25:10.785830 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf" is_pre_election: true
I20260810 06:25:10.785912 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf" is_pre_election: true
I20260810 06:25:10.785972 8092 raft_consensus.cc:2454] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 0.
I20260810 06:25:10.785988 7627 raft_consensus.cc:2454] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8c215936494f9db6ed8f8288d61e83 in term 0.
I20260810 06:25:10.785972 7857 raft_consensus.cc:2454] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8c215936494f9db6ed8f8288d61e83 in term 0.
I20260810 06:25:10.786307 7989 leader_election.cc:302] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 0c8c215936494f9db6ed8f8288d61e83; no voters:
I20260810 06:25:10.786310 7754 leader_election.cc:302] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8c215936494f9db6ed8f8288d61e83, 3b538cbd3e374000898a61d1c985bf1c; no voters:
I20260810 06:25:10.786495 8377 raft_consensus.cc:2781] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:10.786572 8377 raft_consensus.cc:482] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:10.786561 8317 raft_consensus.cc:2781] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:10.786615 8377 raft_consensus.cc:3037] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:10.786615 8317 raft_consensus.cc:482] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:10.786660 8317 raft_consensus.cc:3037] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:10.787887 8377 raft_consensus.cc:504] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.787887 8317 raft_consensus.cc:504] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.788228 8317 leader_election.cc:288] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 election: Requested vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:10.788375 8377 leader_election.cc:288] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 election: Requested vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629)
I20260810 06:25:10.788671 7857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b538cbd3e374000898a61d1c985bf1c"
I20260810 06:25:10.788652 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83"
I20260810 06:25:10.788690 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "0c8c215936494f9db6ed8f8288d61e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf"
I20260810 06:25:10.788652 7626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c55fc4e6b2a4ab1ae477bb7957b20ac" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf"
I20260810 06:25:10.788810 7627 raft_consensus.cc:3037] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:10.788829 7857 raft_consensus.cc:2379] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c8c215936494f9db6ed8f8288d61e83 in current term 1: Already voted for candidate 3b538cbd3e374000898a61d1c985bf1c in this term.
I20260810 06:25:10.788827 8092 raft_consensus.cc:2379] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3b538cbd3e374000898a61d1c985bf1c in current term 1: Already voted for candidate 0c8c215936494f9db6ed8f8288d61e83 in this term.
I20260810 06:25:10.789256 7754 leader_election.cc:302] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3b538cbd3e374000898a61d1c985bf1c; no voters: 02ac2c1a700b441aa7fd74096b860ccf, 0c8c215936494f9db6ed8f8288d61e83
I20260810 06:25:10.789456 8317 raft_consensus.cc:2726] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 06:25:10.789951 7627 raft_consensus.cc:2454] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8c215936494f9db6ed8f8288d61e83 in term 1.
I20260810 06:25:10.790253 7989 leader_election.cc:302] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 0c8c215936494f9db6ed8f8288d61e83; no voters: 3b538cbd3e374000898a61d1c985bf1c
I20260810 06:25:10.790479 8377 raft_consensus.cc:2781] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:10.790711 8377 raft_consensus.cc:686] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8c215936494f9db6ed8f8288d61e83, State: Running, Role: LEADER
I20260810 06:25:10.790874 8377 consensus_queue.cc:237] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.792618 7352 catalog_manager.cc:5495] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "0c8c215936494f9db6ed8f8288d61e83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } health_report { overall_health: HEALTHY } } }
I20260810 06:25:10.847827 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:10.840232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.4.73.3:46175)
0810 06:25:10.844641 (+ 4409us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:10.844650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:10.844661 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:25:10.844699 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:10.844701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:10.844742 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:10.844814 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:10.845117 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:10.845629 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:10.846804 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:10.847021 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:10.847094 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:10.847367 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:10.847434 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:10.847486 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:10.847592 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 06:25:10.847638 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4345,"thread_start_us":123,"threads_started":1}
I20260810 06:25:10.853963 7857 raft_consensus.cc:1261] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Refusing update from remote peer 0c8c215936494f9db6ed8f8288d61e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:10.853963 7627 raft_consensus.cc:1261] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Refusing update from remote peer 0c8c215936494f9db6ed8f8288d61e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:10.854377 8378 consensus_queue.cc:1035] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:10.854559 8377 consensus_queue.cc:1035] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:10.886435 7353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:10.886631 7353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:10.888568 7353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Sending DeleteTablet for 3 replicas of tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac
I20260810 06:25:10.889187 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:10 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:44970
I20260810 06:25:10.889207 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:10 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260810 06:25:10.889464 7812 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:10 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:57034
I20260810 06:25:10.889492 8405 tablet_replica.cc:330] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: stopping tablet replica
I20260810 06:25:10.889637 8405 raft_consensus.cc:2229] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:10.889810 8408 tablet_replica.cc:330] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: stopping tablet replica
I20260810 06:25:10.889837 8405 raft_consensus.cc:2258] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:10.889897 8406 tablet_replica.cc:330] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: stopping tablet replica
I20260810 06:25:10.890018 8406 raft_consensus.cc:2229] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:10.890069 8408 raft_consensus.cc:2229] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:10.890141 8406 raft_consensus.cc:2258] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:10.890172 7353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:25:10.890295 8408 raft_consensus.cc:2258] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 06:25:10.890612 7353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:10.890736 8405 ts_tablet_manager.cc:1905] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:10.890899 8406 ts_tablet_manager.cc:1905] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:10.891286 8408 ts_tablet_manager.cc:1905] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:10.893528 8405 ts_tablet_manager.cc:1918] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:10.893612 8405 log.cc:1196] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/0c55fc4e6b2a4ab1ae477bb7957b20ac
I20260810 06:25:10.893941 8405 ts_tablet_manager.cc:1939] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 0c8c215936494f9db6ed8f8288d61e83: Deleting consensus metadata
I20260810 06:25:10.894224 8406 ts_tablet_manager.cc:1918] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:10.894311 8406 log.cc:1196] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/0c55fc4e6b2a4ab1ae477bb7957b20ac
I20260810 06:25:10.894387 8408 ts_tablet_manager.cc:1918] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:10.894586 8406 ts_tablet_manager.cc:1939] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 02ac2c1a700b441aa7fd74096b860ccf: Deleting consensus metadata
I20260810 06:25:10.894702 7332 catalog_manager.cc:4841] TS 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175): tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac (table test-table [id=64ba3a4398bc485fbd859efe69f5d436]) successfully deleted
I20260810 06:25:10.894505 8408 log.cc:1196] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/0c55fc4e6b2a4ab1ae477bb7957b20ac
I20260810 06:25:10.895167 8408 ts_tablet_manager.cc:1939] T 0c55fc4e6b2a4ab1ae477bb7957b20ac P 3b538cbd3e374000898a61d1c985bf1c: Deleting consensus metadata
I20260810 06:25:10.895366 7332 catalog_manager.cc:4841] TS 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871): tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac (table test-table [id=64ba3a4398bc485fbd859efe69f5d436]) successfully deleted
I20260810 06:25:10.895835 7326 catalog_manager.cc:4841] TS 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629): tablet 0c55fc4e6b2a4ab1ae477bb7957b20ac (table test-table [id=64ba3a4398bc485fbd859efe69f5d436]) successfully deleted
I20260810 06:25:10.896472 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 34f059250dae42f7b271f6e93d1241bd (DEFAULT_TABLE table=test-table [id=b7418e3cf2574401ae90499476464788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:10.896567 7812 tablet_service.cc:1467] Processing CreateTablet for tablet 34f059250dae42f7b271f6e93d1241bd (DEFAULT_TABLE table=test-table [id=b7418e3cf2574401ae90499476464788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:10.896760 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f059250dae42f7b271f6e93d1241bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:10.896780 7812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f059250dae42f7b271f6e93d1241bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:10.896719 8063 tablet_service.cc:1467] Processing CreateTablet for tablet 34f059250dae42f7b271f6e93d1241bd (DEFAULT_TABLE table=test-table [id=b7418e3cf2574401ae90499476464788]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:10.896991 8063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f059250dae42f7b271f6e93d1241bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:10.899124 8409 tablet_bootstrap.cc:492] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf: Bootstrap starting.
I20260810 06:25:10.899266 8414 tablet_bootstrap.cc:492] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: Bootstrap starting.
I20260810 06:25:10.900118 8414 tablet_bootstrap.cc:654] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:10.900215 8409 tablet_bootstrap.cc:654] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:10.900712 8413 tablet_bootstrap.cc:492] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: Bootstrap starting.
I20260810 06:25:10.901554 8413 tablet_bootstrap.cc:654] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:10.902045 8409 tablet_bootstrap.cc:492] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf: No bootstrap required, opened a new log
I20260810 06:25:10.902134 8414 tablet_bootstrap.cc:492] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: No bootstrap required, opened a new log
I20260810 06:25:10.902143 8409 ts_tablet_manager.cc:1397] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 06:25:10.902228 8414 ts_tablet_manager.cc:1397] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:25:10.902626 8409 raft_consensus.cc:348] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.902661 8414 raft_consensus.cc:348] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.902752 8409 raft_consensus.cc:374] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:10.902789 8409 raft_consensus.cc:729] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ac2c1a700b441aa7fd74096b860ccf, State: Initialized, Role: FOLLOWER
I20260810 06:25:10.902793 8414 raft_consensus.cc:374] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:10.902838 8414 raft_consensus.cc:729] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8c215936494f9db6ed8f8288d61e83, State: Initialized, Role: FOLLOWER
I20260810 06:25:10.902961 8409 consensus_queue.cc:260] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.903000 8414 consensus_queue.cc:260] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.903177 8413 tablet_bootstrap.cc:492] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: No bootstrap required, opened a new log
I20260810 06:25:10.903260 8413 ts_tablet_manager.cc:1397] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:25:10.903303 8414 ts_tablet_manager.cc:1428] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:10.903621 8413 raft_consensus.cc:348] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.903748 8413 raft_consensus.cc:374] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:10.903782 8413 raft_consensus.cc:729] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b538cbd3e374000898a61d1c985bf1c, State: Initialized, Role: FOLLOWER
I20260810 06:25:10.903844 8409 ts_tablet_manager.cc:1428] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 06:25:10.903920 8413 consensus_queue.cc:260] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:10.904235 8413 ts_tablet_manager.cc:1428] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:11.001827 8317 raft_consensus.cc:482] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:11.001960 8317 raft_consensus.cc:504] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:11.002408 8317 leader_election.cc:288] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:11.002887 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f059250dae42f7b271f6e93d1241bd" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf" is_pre_election: true
I20260810 06:25:11.002952 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f059250dae42f7b271f6e93d1241bd" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83" is_pre_election: true
I20260810 06:25:11.003022 7627 raft_consensus.cc:2454] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 0.
I20260810 06:25:11.003165 8092 raft_consensus.cc:2454] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 0.
I20260810 06:25:11.003386 7754 leader_election.cc:302] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02ac2c1a700b441aa7fd74096b860ccf, 3b538cbd3e374000898a61d1c985bf1c; no voters:
I20260810 06:25:11.003566 8317 raft_consensus.cc:2781] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:11.003628 8317 raft_consensus.cc:482] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:11.003671 8317 raft_consensus.cc:3037] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:11.004710 8317 raft_consensus.cc:504] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:11.005040 8317 leader_election.cc:288] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 election: Requested vote from peers 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871), 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175)
I20260810 06:25:11.005415 7627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f059250dae42f7b271f6e93d1241bd" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02ac2c1a700b441aa7fd74096b860ccf"
I20260810 06:25:11.005446 8092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f059250dae42f7b271f6e93d1241bd" candidate_uuid: "3b538cbd3e374000898a61d1c985bf1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c215936494f9db6ed8f8288d61e83"
I20260810 06:25:11.005556 7627 raft_consensus.cc:3037] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:11.005556 8092 raft_consensus.cc:3037] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:11.006706 8092 raft_consensus.cc:2454] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 1.
I20260810 06:25:11.006706 7627 raft_consensus.cc:2454] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b538cbd3e374000898a61d1c985bf1c in term 1.
I20260810 06:25:11.007002 7754 leader_election.cc:302] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8c215936494f9db6ed8f8288d61e83, 3b538cbd3e374000898a61d1c985bf1c; no voters:
I20260810 06:25:11.007174 8317 raft_consensus.cc:2781] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:11.007264 8317 raft_consensus.cc:686] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 1 LEADER]: Becoming Leader. State: Replica: 3b538cbd3e374000898a61d1c985bf1c, State: Running, Role: LEADER
I20260810 06:25:11.007406 8317 consensus_queue.cc:237] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } }
I20260810 06:25:11.009122 7352 catalog_manager.cc:5495] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2). New cstate: current_term: 1 leader_uuid: "3b538cbd3e374000898a61d1c985bf1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b538cbd3e374000898a61d1c985bf1c" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 41629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 } health_report { overall_health: UNKNOWN } } }
I20260810 06:25:11.027853 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:11.024554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.4.73.2:41629)
0810 06:25:11.024881 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:11.024890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:11.024901 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:25:11.024949 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:11.024951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:11.024992 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:11.025090 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:11.025339 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:11.025885 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:11.026759 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:11.026996 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:11.027104 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:11.027369 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:11.027453 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:11.027544 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:11.027635 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 06:25:11.027689 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":146,"threads_started":1}
I20260810 06:25:11.035997 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:11.033297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:60941 (local address 127.4.73.2:41629)
0810 06:25:11.033418 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:11.033421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:11.033484 (+ 63us) server_negotiation.cc:407] Connection header received
0810 06:25:11.033598 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:11.033600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:11.033639 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:11.033699 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:11.033933 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:11.034398 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:11.035168 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:11.035374 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:11.035447 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:11.035663 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:11.035717 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:11.035769 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:11.035848 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 06:25:11.035890 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":55}
I20260810 06:25:11.036021 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:11.033158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:41629 (local address 127.4.73.2:60941)
0810 06:25:11.033387 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:11.033400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:11.033500 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:11.033723 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:11.033725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:11.033732 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:11.033843 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:11.033847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:11.034488 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:11.034490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:11.035068 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:11.035076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:11.035467 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:11.035470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:11.035540 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:11.035564 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:11.035787 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:11.035791 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:25:11.035835 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 06:25:11.035896 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260810 06:25:11.041590 7627 raft_consensus.cc:1261] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Refusing update from remote peer 3b538cbd3e374000898a61d1c985bf1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:11.041623 8092 raft_consensus.cc:1261] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Refusing update from remote peer 3b538cbd3e374000898a61d1c985bf1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:11.042003 8385 consensus_queue.cc:1035] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8c215936494f9db6ed8f8288d61e83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:11.042151 8317 consensus_queue.cc:1035] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ac2c1a700b441aa7fd74096b860ccf" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 06:25:11.062038 8380 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 06:25:12.617982 7353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:12.618144 7353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:12.620528 7353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd6773bcde6d4e3e8923fa6dffbe65fa: Sending DeleteTablet for 3 replicas of tablet 34f059250dae42f7b271f6e93d1241bd
I20260810 06:25:12.621090 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f059250dae42f7b271f6e93d1241bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:12 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:46142
I20260810 06:25:12.621212 8063 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f059250dae42f7b271f6e93d1241bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:12 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:44970
I20260810 06:25:12.621399 7812 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f059250dae42f7b271f6e93d1241bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:12 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:57034
I20260810 06:25:12.621461 8461 tablet_replica.cc:330] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf: stopping tablet replica
I20260810 06:25:12.621594 8461 raft_consensus.cc:2229] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:12.621642 8466 tablet_replica.cc:330] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: stopping tablet replica
I20260810 06:25:12.621814 8467 tablet_replica.cc:330] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: stopping tablet replica
I20260810 06:25:12.621842 8461 raft_consensus.cc:2258] T 34f059250dae42f7b271f6e93d1241bd P 02ac2c1a700b441aa7fd74096b860ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:12.621939 8466 raft_consensus.cc:2229] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:12.621990 8467 raft_consensus.cc:2229] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:12.622123 8466 raft_consensus.cc:2258] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:12.622206 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7479
I20260810 06:25:12.626884 8467 raft_consensus.cc:2258] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:12.629081 8467 ts_tablet_manager.cc:1905] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:12.632215 8467 ts_tablet_manager.cc:1918] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:25:12.632377 8467 log.cc:1196] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/34f059250dae42f7b271f6e93d1241bd
I20260810 06:25:12.632692 8467 ts_tablet_manager.cc:1939] T 34f059250dae42f7b271f6e93d1241bd P 3b538cbd3e374000898a61d1c985bf1c: Deleting consensus metadata
I20260810 06:25:12.637658 7326 catalog_manager.cc:4841] TS 3b538cbd3e374000898a61d1c985bf1c (127.4.73.2:41629): tablet 34f059250dae42f7b271f6e93d1241bd (table test-table [id=b7418e3cf2574401ae90499476464788]) successfully deleted
I20260810 06:25:12.666167 8466 ts_tablet_manager.cc:1905] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:12.669795 8466 ts_tablet_manager.cc:1918] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 06:25:12.669912 8466 log.cc:1196] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/34f059250dae42f7b271f6e93d1241bd
I20260810 06:25:12.670188 8466 ts_tablet_manager.cc:1939] T 34f059250dae42f7b271f6e93d1241bd P 0c8c215936494f9db6ed8f8288d61e83: Deleting consensus metadata
I20260810 06:25:12.671128 7332 catalog_manager.cc:4841] TS 0c8c215936494f9db6ed8f8288d61e83 (127.4.73.3:46175): tablet 34f059250dae42f7b271f6e93d1241bd (table test-table [id=b7418e3cf2574401ae90499476464788]) successfully deleted
W20260810 06:25:12.674013 7332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:40871 (error 108)
I20260810 06:25:12.674271 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7718
I20260810 06:25:12.680501 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:12.679435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40871 (local address 127.0.0.1:48194)
0810 06:25:12.680224 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:12.680363 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":569,"threads_started":1}
W20260810 06:25:12.680655 7332 catalog_manager.cc:4584] TS 02ac2c1a700b441aa7fd74096b860ccf (127.4.73.1:40871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34f059250dae42f7b271f6e93d1241bd: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40871: connect: Connection refused (error 111)
I20260810 06:25:12.687402 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7960
I20260810 06:25:12.704061 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7299
2026-08-10T06:25:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7026 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:12 dist-test-slave-45dx krb5kdc[8473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:12 dist-test-slave-45dx krb5kdc[8473](info): set up 2 sockets
Aug 10 06:25:12 dist-test-slave-45dx krb5kdc[8473](info): commencing operation
krb5kdc: starting...
W20260810 06:25:13.968515 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:14.092959 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:42007
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:14.202915 8507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:14.203161 8507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:14.203199 8507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:14.207063 8507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:14.207111 8507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:14.207129 8507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:14.207145 8507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:14.207159 8507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:14.212109 8507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:14.213037 8507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:14.214053 8507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:14.222146 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:14.222172 8520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:14.222348 8507 server_base.cc:1034] running on GCE node
W20260810 06:25:14.222141 8523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:14.222923 8507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:14.223879 8507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:14.225036 8507 hybrid_clock.cc:648] HybridClock initialized: now 1786343114225008 us; error 40 us; skew 500 ppm
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:14.227695 8507 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:14.228580 8507 webserver.cc:460] Webserver started at http://127.4.73.62:36707/ using document root <none> and password file <none>
I20260810 06:25:14.229090 8507 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:14.229149 8507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:14.229380 8507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:14.231305 8507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "9a31d5676cf14f2296b9b2e98ff7452f"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "d835f2981f8e4253938c2e88dd14dad6"
server_key_iv: "c69b050d41a9c1fe1caf7da2911bc3f8"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.232046 8507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "9a31d5676cf14f2296b9b2e98ff7452f"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "d835f2981f8e4253938c2e88dd14dad6"
server_key_iv: "c69b050d41a9c1fe1caf7da2911bc3f8"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.236516 8507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:25:14.238749 8536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.239810 8507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:14.239964 8507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "9a31d5676cf14f2296b9b2e98ff7452f"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "d835f2981f8e4253938c2e88dd14dad6"
server_key_iv: "c69b050d41a9c1fe1caf7da2911bc3f8"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.240070 8507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:14.267063 8507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:14.271188 8507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:14.271407 8507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:14.278275 8507 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:42007
I20260810 06:25:14.278306 8615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:42007 every 8 connection(s)
I20260810 06:25:14.279181 8507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:14.281989 8616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:14.287483 8616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f: Bootstrap starting.
I20260810 06:25:14.288774 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8507
I20260810 06:25:14.288905 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:14.289165 4388 external_mini_cluster.cc:1428] Setting key f21fd8b235a46879b9a604a2f73ef0fc
I20260810 06:25:14.290226 8616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:14.291085 8616 log.cc:824] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:14.293162 8616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f: No bootstrap required, opened a new log
I20260810 06:25:14.296962 8616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42007 } }
I20260810 06:25:14.297215 8616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:14.297304 8616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a31d5676cf14f2296b9b2e98ff7452f, State: Initialized, Role: FOLLOWER
I20260810 06:25:14.297857 8616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42007 } }
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:14.298038 8616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:14.298122 8616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:14.298228 8616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:14.299378 8616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42007 } }
I20260810 06:25:14.299803 8616 leader_election.cc:302] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a31d5676cf14f2296b9b2e98ff7452f; no voters:
I20260810 06:25:14.300164 8616 leader_election.cc:288] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:14.300287 8624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:14.300539 8624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [term 1 LEADER]: Becoming Leader. State: Replica: 9a31d5676cf14f2296b9b2e98ff7452f, State: Running, Role: LEADER
I20260810 06:25:14.300876 8624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42007 } }
I20260810 06:25:14.301568 8616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:14.302142 8630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a31d5676cf14f2296b9b2e98ff7452f. Latest consensus state: current_term: 1 leader_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42007 } } }
I20260810 06:25:14.302291 8630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:14.302611 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a31d5676cf14f2296b9b2e98ff7452f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 42007 } } }
I20260810 06:25:14.302695 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:14.302973 8634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:14.305526 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:14.290312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.4.73.62:42007)
0810 06:25:14.290772 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:14.290787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:14.290822 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:25:14.291550 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:14.291576 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:14.291985 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:14.292368 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:14.293195 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.294201 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.295086 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.295587 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.295769 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:14.298700 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:14.298705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:14.298718 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:14.298744 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:14.301036 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:14.301580 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:14.301586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:14.301591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:14.301727 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:14.302121 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:14.302125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:14.302126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:14.302743 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:14.302954 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:14.304166 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0810 06:25:14.304551 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":140,"threads_started":1}
I20260810 06:25:14.306152 8634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:14.311882 8634 catalog_manager.cc:1347] Generated new cluster ID: 0dae3608e3a849e6b6b15c84ed0f2425
I20260810 06:25:14.311954 8634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:14.319687 8634 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:14.320864 8634 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:14.334497 8634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a31d5676cf14f2296b9b2e98ff7452f: Generated new TSK 0
I20260810 06:25:14.335373 8634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:14.365598 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42007
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:14.478231 8670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:14.478506 8670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:14.478574 8670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:14.482556 8670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:14.482640 8670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:14.482748 8670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:14.487802 8670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:14.488763 8670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:14.489744 8670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:25:14.499514 8670 server_base.cc:1034] running on GCE node
W20260810 06:25:14.499471 8680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:14.499445 8679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:14.499464 8688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:14.500170 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:14.500818 8670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:14.501978 8670 hybrid_clock.cc:648] HybridClock initialized: now 1786343114501960 us; error 31 us; skew 500 ppm
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:14.504977 8670 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:14.505775 8670 webserver.cc:460] Webserver started at http://127.4.73.1:35311/ using document root <none> and password file <none>
I20260810 06:25:14.506276 8670 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:14.506335 8670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:14.506495 8670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:14.508543 8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "d5f9f9995dfe42daa6ee1a50d078a811"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "6d645d0a69d89d04d328e9b6fa1008db"
server_key_iv: "49db087aea83ba7c7e722dbb851801fd"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.509080 8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f9f9995dfe42daa6ee1a50d078a811"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "6d645d0a69d89d04d328e9b6fa1008db"
server_key_iv: "49db087aea83ba7c7e722dbb851801fd"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.512656 8670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:14.514972 8701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.516144 8670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:14.516280 8670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "d5f9f9995dfe42daa6ee1a50d078a811"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "6d645d0a69d89d04d328e9b6fa1008db"
server_key_iv: "49db087aea83ba7c7e722dbb851801fd"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.516433 8670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:14.531608 8670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:14.534788 8670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:14.534994 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:14.535594 8670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:14.536487 8670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:14.536537 8670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.536616 8670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:14.536648 8670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.545513 8670 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:37655
I20260810 06:25:14.545589 8888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:37655 every 8 connection(s)
I20260810 06:25:14.546416 8670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:14.552080 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8670
I20260810 06:25:14.552189 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:14.552491 4388 external_mini_cluster.cc:1428] Setting key 474e772043f2b72ef902c39cd03a22f1
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:14.561249 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:14.548158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:50337 (local address 127.4.73.62:42007)
0810 06:25:14.548289 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:14.548301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:14.549142 (+ 841us) server_negotiation.cc:407] Connection header received
0810 06:25:14.550527 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:14.550531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:14.550607 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:14.550713 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:14.551846 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.552621 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.554191 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.554444 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.554558 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:14.557166 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:14.557173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:14.557179 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:14.557214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:14.558947 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:14.559487 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:14.559493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:14.559496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:14.559605 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:14.559963 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:14.559970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:14.559975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:14.560250 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:14.560392 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:14.560881 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 06:25:14.561031 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:25:14.561905 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:14.548558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42007 (local address 127.4.73.1:50337)
0810 06:25:14.548996 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:14.549038 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:14.550290 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:14.550889 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:14.550896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:14.551227 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:14.551680 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:14.551694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.552749 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.552753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:14.554055 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:14.554066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.554598 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.554602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:14.554781 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:14.555420 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:14.555436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:14.556995 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:14.559079 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:14.559087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:14.559103 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:14.559365 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:14.559704 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:14.559706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:14.559707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:14.559858 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:14.560384 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:14.560389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:14.560628 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 06:25:14.560779 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 06:25:14.561021 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260810 06:25:14.563129 8889 heartbeater.cc:347] Connected to a master server at 127.4.73.62:42007
I20260810 06:25:14.563402 8889 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:25:14.564020 8889 heartbeater.cc:511] Master 127.4.73.62:42007 requested a full tablet report, sending...
I20260810 06:25:14.565685 8559 ts_manager.cc:194] Registered new tserver with Master: d5f9f9995dfe42daa6ee1a50d078a811 (127.4.73.1:37655)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:14.566915 8559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:50337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:14.581477 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42007
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:14.691306 8904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:14.691533 8904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:14.691570 8904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:14.695637 8904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:14.695705 8904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:14.695806 8904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:14.701136 8904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:14.702214 8904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:14.703298 8904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:14.713135 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:14.713135 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:14.713204 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:14.713372 8904 server_base.cc:1034] running on GCE node
I20260810 06:25:14.713898 8904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:14.714485 8904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:14.715651 8904 hybrid_clock.cc:648] HybridClock initialized: now 1786343114715631 us; error 31 us; skew 500 ppm
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:14.718708 8904 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:14.719697 8904 webserver.cc:460] Webserver started at http://127.4.73.2:44997/ using document root <none> and password file <none>
I20260810 06:25:14.720283 8904 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:14.720392 8904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:14.720608 8904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:14.722620 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "3a69a0bec2a041a0bf6de149ed031c9f"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "a1246575b526dbdc72e96c7cf6030cb7"
server_key_iv: "26ec2d95bb54241975e657fbafe0c153"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.723186 8904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "3a69a0bec2a041a0bf6de149ed031c9f"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "a1246575b526dbdc72e96c7cf6030cb7"
server_key_iv: "26ec2d95bb54241975e657fbafe0c153"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.726723 8904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 06:25:14.729059 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.730228 8904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 06:25:14.730365 8904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "3a69a0bec2a041a0bf6de149ed031c9f"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "a1246575b526dbdc72e96c7cf6030cb7"
server_key_iv: "26ec2d95bb54241975e657fbafe0c153"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.730468 8904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:14.756410 8904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:14.759239 8904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:14.759461 8904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:14.760047 8904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:14.760881 8904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:14.760949 8904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.761016 8904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:14.761056 8904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.769649 8904 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:41691
I20260810 06:25:14.769692 9125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:41691 every 8 connection(s)
I20260810 06:25:14.770500 8904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:14.778040 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8904
I20260810 06:25:14.778182 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:14.778507 4388 external_mini_cluster.cc:1428] Setting key 8b0e4f5f9f0cf1f658c34656dc29269d
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:14.784641 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:14.772191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:34051 (local address 127.4.73.62:42007)
0810 06:25:14.772346 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:14.772366 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:14.773119 (+ 753us) server_negotiation.cc:407] Connection header received
0810 06:25:14.774466 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:14.774470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:14.774550 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:14.774643 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:14.775791 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.776381 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.777530 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.777829 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.777942 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:14.780742 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:14.780748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:14.780754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:14.780789 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:14.782433 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:14.783059 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:14.783061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:14.783063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:14.783140 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:14.783543 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:14.783545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:14.783546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:14.783712 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:14.783789 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:14.784424 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 06:25:14.784524 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:25:14.785871 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:14.772548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42007 (local address 127.4.73.2:34051)
0810 06:25:14.772947 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:14.772998 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:14.774245 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:14.774816 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:14.774822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:14.775170 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:14.775641 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:14.775654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.776508 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.776513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:14.777394 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:14.777403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.777985 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.777988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:14.778143 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:14.778857 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:14.778872 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:14.780579 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:14.782551 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:14.782560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:14.782574 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:14.782953 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:14.783225 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:14.783228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:14.783229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:14.783422 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:14.783819 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:14.783828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:14.784138 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 06:25:14.784394 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 06:25:14.784745 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":94,"threads_started":1}
I20260810 06:25:14.787052 9126 heartbeater.cc:347] Connected to a master server at 127.4.73.62:42007
I20260810 06:25:14.787328 9126 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:14.787976 9126 heartbeater.cc:511] Master 127.4.73.62:42007 requested a full tablet report, sending...
I20260810 06:25:14.789178 8559 ts_manager.cc:194] Registered new tserver with Master: 3a69a0bec2a041a0bf6de149ed031c9f (127.4.73.2:41691)
I20260810 06:25:14.789757 8559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:34051
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:14.807258 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42007
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:14.914176 9133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:14.914434 9133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:14.914505 9133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:14.918362 9133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:14.918438 9133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:14.918522 9133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:14.923861 9133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:42007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:14.924824 9133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:14.925781 9133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:14.934679 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:14.934688 9152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:14.934687 9150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:14.934885 9133 server_base.cc:1034] running on GCE node
I20260810 06:25:14.935442 9133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:14.936014 9133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:14.937194 9133 hybrid_clock.cc:648] HybridClock initialized: now 1786343114937155 us; error 51 us; skew 500 ppm
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:14.939759 9133 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:14.940589 9133 webserver.cc:460] Webserver started at http://127.4.73.3:37105/ using document root <none> and password file <none>
I20260810 06:25:14.941100 9133 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:14.941159 9133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:14.941323 9133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:14.943167 9133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "4c768d2d19a040629c96f81625c37573"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "0a48278e149fa9ff2518d668a0f50049"
server_key_iv: "5cfbddd769f2ede6c7fb4dfc73254682"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.943717 9133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "4c768d2d19a040629c96f81625c37573"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "0a48278e149fa9ff2518d668a0f50049"
server_key_iv: "5cfbddd769f2ede6c7fb4dfc73254682"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.947235 9133 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:25:14.949472 9161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.950640 9133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:14.950774 9133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "4c768d2d19a040629c96f81625c37573"
format_stamp: "Formatted at 2026-08-10 06:25:14 on dist-test-slave-45dx"
server_key: "0a48278e149fa9ff2518d668a0f50049"
server_key_iv: "5cfbddd769f2ede6c7fb4dfc73254682"
server_key_version: "encryptionkey@0"
I20260810 06:25:14.950887 9133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:14.973958 9133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:14.978385 9133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:14.978591 9133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:14.979363 9133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:14.980472 9133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:14.980540 9133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.980594 9133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:14.980625 9133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:14.991593 9133 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:42249
I20260810 06:25:14.991619 9357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:42249 every 8 connection(s)
I20260810 06:25:14.992588 9133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:14.994210 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9133
I20260810 06:25:14.994345 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:14.994629 4388 external_mini_cluster.cc:1428] Setting key 20620da43eb583d50f32fc428adf2a63
Aug 10 06:25:14 dist-test-slave-45dx krb5kdc[8473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343114, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:15.006633 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:14.995771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:37597 (local address 127.4.73.62:42007)
0810 06:25:14.995921 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:14.995925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:14.995940 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:25:14.997054 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:14.997057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:14.997102 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:14.997171 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:14.998290 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.998829 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.999920 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:15.000144 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:15.000219 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:15.002809 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:15.002813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:15.002815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:15.002838 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:15.004263 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:15.004800 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:15.004802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:15.004804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:15.004884 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:15.005364 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:15.005370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:15.005380 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:15.005667 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:15.005808 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:15.006400 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 06:25:15.006507 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:25:15.007642 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:14.994818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:42007 (local address 127.4.73.3:37597)
0810 06:25:14.995284 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:14.995335 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:14.996873 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:14.997329 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:14.997335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:14.997697 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:14.998152 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:14.998165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:14.998934 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:14.998936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:14.999817 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:14.999826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:15.000272 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:15.000275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:15.000474 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:15.001052 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:15.001068 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:15.002588 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:15.004439 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:15.004445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:15.004456 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:15.004712 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:15.004997 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:15.005001 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:15.005003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:15.005236 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:15.005794 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:15.005802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:15.006160 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 06:25:15.006328 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 06:25:15.006598 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
I20260810 06:25:15.009114 9358 heartbeater.cc:347] Connected to a master server at 127.4.73.62:42007
I20260810 06:25:15.009367 9358 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:15.009883 9358 heartbeater.cc:511] Master 127.4.73.62:42007 requested a full tablet report, sending...
I20260810 06:25:15.010948 8559 ts_manager.cc:194] Registered new tserver with Master: 4c768d2d19a040629c96f81625c37573 (127.4.73.3:42249)
I20260810 06:25:15.011488 8559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:37597
I20260810 06:25:15.020941 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 06:25:15.037772 8621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:15.035973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.4.73.62:42007)
0810 06:25:15.036247 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:15.036252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:15.036453 (+ 201us) server_negotiation.cc:407] Connection header received
0810 06:25:15.036647 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:15.036650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:15.036708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:15.036792 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:15.037653 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54180: BlockingRecv error: recv got EOF from 127.0.0.1:54180 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
I20260810 06:25:15.039098 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8670
I20260810 06:25:15.063509 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8904
I20260810 06:25:15.085474 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9133
I20260810 06:25:15.098650 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8507
2026-08-10T06:25:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2391 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:15 dist-test-slave-45dx krb5kdc[9376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:15 dist-test-slave-45dx krb5kdc[9376](info): set up 2 sockets
Aug 10 06:25:15 dist-test-slave-45dx krb5kdc[9376](info): commencing operation
krb5kdc: starting...
W20260810 06:25:16.374747 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:16 dist-test-slave-45dx krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:16.495046 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:35623
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:16.611372 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:16.611671 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:16.611745 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:16.615763 9397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:16.615860 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:16.615898 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:16.615950 9397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:16.615988 9397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:16.621167 9397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:16.622177 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:16.623278 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:16.631693 9407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:16.631697 9410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:16.631734 9408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:16.631920 9397 server_base.cc:1034] running on GCE node
I20260810 06:25:16.632614 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:16.633659 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:16.634842 9397 hybrid_clock.cc:648] HybridClock initialized: now 1786343116634820 us; error 43 us; skew 500 ppm
Aug 10 06:25:16 dist-test-slave-45dx krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343116, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:16.637559 9397 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:16.638319 9397 webserver.cc:460] Webserver started at http://127.4.73.62:33279/ using document root <none> and password file <none>
I20260810 06:25:16.638792 9397 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:16.638857 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:16.639014 9397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:16.640983 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5"
format_stamp: "Formatted at 2026-08-10 06:25:16 on dist-test-slave-45dx"
server_key: "d50ff3e3450ba4a7325543be987d788f"
server_key_iv: "93cf39660b885e3037f90fc660c47224"
server_key_version: "encryptionkey@0"
I20260810 06:25:16.641489 9397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5"
format_stamp: "Formatted at 2026-08-10 06:25:16 on dist-test-slave-45dx"
server_key: "d50ff3e3450ba4a7325543be987d788f"
server_key_iv: "93cf39660b885e3037f90fc660c47224"
server_key_version: "encryptionkey@0"
I20260810 06:25:16.644955 9397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:25:16.647166 9423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:16.648351 9397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:16.648492 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5"
format_stamp: "Formatted at 2026-08-10 06:25:16 on dist-test-slave-45dx"
server_key: "d50ff3e3450ba4a7325543be987d788f"
server_key_iv: "93cf39660b885e3037f90fc660c47224"
server_key_version: "encryptionkey@0"
I20260810 06:25:16.648589 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:16.665518 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:16.668056 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:16.668246 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:16.674891 9397 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:35623
I20260810 06:25:16.675247 9510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:35623 every 8 connection(s)
I20260810 06:25:16.675801 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:16.678933 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:16.680538 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9397
I20260810 06:25:16.680680 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:16.680949 4388 external_mini_cluster.cc:1428] Setting key ff25d9c96f218e8d187f6994b25752a5
I20260810 06:25:16.685623 9511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5: Bootstrap starting.
I20260810 06:25:16.688095 9511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:16.688930 9511 log.cc:824] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:25:16 dist-test-slave-45dx krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:16.690907 9511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5: No bootstrap required, opened a new log
I20260810 06:25:16.695675 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:16.682154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.4.73.62:35623)
0810 06:25:16.682562 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:16.682585 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:16.682625 (+ 40us) server_negotiation.cc:407] Connection header received
0810 06:25:16.683432 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:16.683457 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:16.683841 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:16.684184 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:16.684966 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:16.686348 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:16.687217 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:16.687878 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:16.688055 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:16.690496 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:16.690502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:16.690517 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:16.690548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:16.692623 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:16.693062 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:16.693067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:16.693072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:16.693236 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:16.693549 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:16.693553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:16.693556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:16.694049 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:16.694236 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:16.694466 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 06:25:16.694764 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":105,"threads_started":1}
I20260810 06:25:16.695575 9511 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35623 } }
I20260810 06:25:16.695847 9511 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:16.695936 9511 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ad3711890f4eb9b8ae9f10ea571fe5, State: Initialized, Role: FOLLOWER
I20260810 06:25:16.696446 9511 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35623 } }
I20260810 06:25:16.696601 9511 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:16.696668 9511 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:16.696775 9511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:16.697767 9511 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35623 } }
I20260810 06:25:16.698123 9511 leader_election.cc:302] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6ad3711890f4eb9b8ae9f10ea571fe5; no voters:
I20260810 06:25:16.698458 9511 leader_election.cc:288] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:16.698526 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:25:16.698549 9523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:16.698751 9523 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [term 1 LEADER]: Becoming Leader. State: Replica: f6ad3711890f4eb9b8ae9f10ea571fe5, State: Running, Role: LEADER
I20260810 06:25:16.699029 9523 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35623 } }
I20260810 06:25:16.699790 9511 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:16.700443 9527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6ad3711890f4eb9b8ae9f10ea571fe5. Latest consensus state: current_term: 1 leader_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35623 } } }
I20260810 06:25:16.700523 9524 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6ad3711890f4eb9b8ae9f10ea571fe5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35623 } } }
I20260810 06:25:16.700584 9527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:16.700584 9524 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5 [sys.catalog]: This master's current role is: LEADER
W20260810 06:25:16.705051 9552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:25:16.705137 9552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:25:16.705205 9534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:16.705907 9534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:16.710882 9534 catalog_manager.cc:1347] Generated new cluster ID: 91ea87e402cb4f4a9a0abc5f8c2b6eae
I20260810 06:25:16.710940 9534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:16.720794 9534 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:16.722139 9534 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:16.727043 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:16.721133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42122 (local address 127.4.73.62:35623)
0810 06:25:16.721253 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:16.721258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:16.721515 (+ 257us) server_negotiation.cc:407] Connection header received
0810 06:25:16.721645 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:16.721649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:16.721697 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:16.721805 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:16.722399 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:16.723149 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:16.724476 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:16.724832 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:16.724896 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:16.725953 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:16.725957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:16.725960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:16.726555 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:16.726593 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:16.726613 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:16.726705 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:16.726832 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 06:25:16.726899 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 06:25:16.730581 9534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6ad3711890f4eb9b8ae9f10ea571fe5: Generated new TSK 0
I20260810 06:25:16.731456 9534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:25:16.736758 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9397
2026-08-10T06:25:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1638 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:16 dist-test-slave-45dx krb5kdc[9572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:16 dist-test-slave-45dx krb5kdc[9572](info): set up 2 sockets
Aug 10 06:25:16 dist-test-slave-45dx krb5kdc[9572](info): commencing operation
krb5kdc: starting...
W20260810 06:25:18.004428 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:18.127787 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38069
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:38069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:18.241338 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:18.241606 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:18.241652 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:18.245496 9594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:18.245545 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:18.245563 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:18.245577 9594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:18.245591 9594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:18.251754 9594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:38069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:18.252801 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:18.253775 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:18.262487 9603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:18.262610 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:18.262494 9610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:18.263058 9594 server_base.cc:1034] running on GCE node
I20260810 06:25:18.263752 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:18.265340 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:18.266531 9594 hybrid_clock.cc:648] HybridClock initialized: now 1786343118266510 us; error 40 us; skew 500 ppm
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:18.273247 9594 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:18.274232 9594 webserver.cc:460] Webserver started at http://127.4.73.62:38303/ using document root <none> and password file <none>
I20260810 06:25:18.274756 9594 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:18.274814 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:18.274976 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:18.277078 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "8b41851ce084415791b0de30200f56d3"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "a7115a32f10dc294faa5e93439b02a13"
server_key_iv: "846044be47ef2d9b251858f8266d47b0"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.277586 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "8b41851ce084415791b0de30200f56d3"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "a7115a32f10dc294faa5e93439b02a13"
server_key_iv: "846044be47ef2d9b251858f8266d47b0"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.281122 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:25:18.283375 9623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.284713 9594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:18.284842 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "8b41851ce084415791b0de30200f56d3"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "a7115a32f10dc294faa5e93439b02a13"
server_key_iv: "846044be47ef2d9b251858f8266d47b0"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.284940 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:18.304309 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:18.307261 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:18.307471 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:18.314186 9594 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:38069
I20260810 06:25:18.314479 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:38069 every 8 connection(s)
I20260810 06:25:18.315217 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:18.318055 9708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:18.323625 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3: Bootstrap starting.
I20260810 06:25:18.324940 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9594
I20260810 06:25:18.325052 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:18.325357 4388 external_mini_cluster.cc:1428] Setting key 8d3b7018db27e8bed08fc31e139a0039
I20260810 06:25:18.326293 9708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:18.327137 9708 log.cc:824] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:18.329064 9708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3: No bootstrap required, opened a new log
I20260810 06:25:18.332326 9708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41851ce084415791b0de30200f56d3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38069 } }
I20260810 06:25:18.332616 9708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:18.332703 9708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b41851ce084415791b0de30200f56d3, State: Initialized, Role: FOLLOWER
I20260810 06:25:18.333138 9708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41851ce084415791b0de30200f56d3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38069 } }
I20260810 06:25:18.333290 9708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:18.333405 9708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:18.333501 9708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:18.334479 9708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41851ce084415791b0de30200f56d3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38069 } }
I20260810 06:25:18.334856 9708 leader_election.cc:302] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b41851ce084415791b0de30200f56d3; no voters:
I20260810 06:25:18.335240 9708 leader_election.cc:288] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:18.335345 9717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:18.335585 9717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [term 1 LEADER]: Becoming Leader. State: Replica: 8b41851ce084415791b0de30200f56d3, State: Running, Role: LEADER
I20260810 06:25:18.335898 9717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41851ce084415791b0de30200f56d3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38069 } }
I20260810 06:25:18.336649 9708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:18.338445 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b41851ce084415791b0de30200f56d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41851ce084415791b0de30200f56d3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38069 } } }
I20260810 06:25:18.338615 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:18.338873 9719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b41851ce084415791b0de30200f56d3. Latest consensus state: current_term: 1 leader_uuid: "8b41851ce084415791b0de30200f56d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41851ce084415791b0de30200f56d3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38069 } } }
I20260810 06:25:18.338975 9719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:18.339032 9728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:18.340967 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:18.326510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.4.73.62:38069)
0810 06:25:18.326976 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:18.326992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:18.327024 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:25:18.327818 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:18.327844 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:18.328178 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:18.328553 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:18.329364 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.330324 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.331155 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.331605 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.331769 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:18.334593 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:18.334598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:18.334647 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:18.334681 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:18.337207 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:18.337756 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:18.337761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:18.337767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:18.337910 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:18.338348 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:18.338354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:18.338358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:18.338904 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:18.339155 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:18.339615 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 06:25:18.339934 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":133,"threads_started":1}
I20260810 06:25:18.342070 9728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:18.347705 9728 catalog_manager.cc:1347] Generated new cluster ID: 3b3ec94feb8b43e1b6a2ce86191d0230
I20260810 06:25:18.347774 9728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:18.364694 9728 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:18.365658 9728 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:18.371575 9728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b41851ce084415791b0de30200f56d3: Generated new TSK 0
I20260810 06:25:18.372217 9728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:18.407245 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38069
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:18.520586 9752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:18.520838 9752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:18.520889 9752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:18.524994 9752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:18.525046 9752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:18.525141 9752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:18.530352 9752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:18.531337 9752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:18.532385 9752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:18.542029 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:18.542021 9763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:18.542096 9752 server_base.cc:1034] running on GCE node
W20260810 06:25:18.542078 9771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:18.542676 9752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:18.543273 9752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:18.544461 9752 hybrid_clock.cc:648] HybridClock initialized: now 1786343118544438 us; error 32 us; skew 500 ppm
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:18.547096 9752 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:18.547901 9752 webserver.cc:460] Webserver started at http://127.4.73.1:41079/ using document root <none> and password file <none>
I20260810 06:25:18.548516 9752 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:18.548606 9752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:18.548836 9752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:18.550768 9752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "63c21c61597c4296bfbcf79196375ec9"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "d714db81b2db2b5c5e9d9e933777e250"
server_key_iv: "655683892f5d5955aecc1efe87d64c73"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.551330 9752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "63c21c61597c4296bfbcf79196375ec9"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "d714db81b2db2b5c5e9d9e933777e250"
server_key_iv: "655683892f5d5955aecc1efe87d64c73"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.554951 9752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:25:18.557310 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.558432 9752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:18.558561 9752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "63c21c61597c4296bfbcf79196375ec9"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "d714db81b2db2b5c5e9d9e933777e250"
server_key_iv: "655683892f5d5955aecc1efe87d64c73"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.558670 9752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:18.568249 9752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:18.570901 9752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:18.571097 9752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:18.571610 9752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:18.572398 9752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:18.572463 9752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.572530 9752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:18.572566 9752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.581040 9752 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:36345
I20260810 06:25:18.581077 9971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:36345 every 8 connection(s)
I20260810 06:25:18.581897 9752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:18.584331 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9752
I20260810 06:25:18.584463 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:18.584794 4388 external_mini_cluster.cc:1428] Setting key fd3ef1ab98f1017674b7b4b91d5dc87a
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:25:18.595925 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:18.583674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:53133 (local address 127.4.73.62:38069)
0810 06:25:18.583818 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:18.583822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:18.584755 (+ 933us) server_negotiation.cc:407] Connection header received
0810 06:25:18.586387 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:18.586390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:18.586442 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:18.586520 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:18.587655 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.588394 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.589654 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.589897 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.590005 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:18.592388 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:18.592391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:18.592394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:18.592419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:18.593821 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:18.594371 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:18.594375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:18.594376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:18.594464 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:18.594811 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:18.594815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:18.594817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:18.594999 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:18.595108 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:18.595611 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 06:25:18.595732 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:25:18.596786 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:18.584058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38069 (local address 127.4.73.1:53133)
0810 06:25:18.584618 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:18.584661 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:18.586192 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:18.586685 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:18.586691 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:18.587054 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:18.587513 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:18.587526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.588525 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.588529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:18.589526 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:18.589536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.590046 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.590050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:18.590211 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:18.590817 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:18.590833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:18.592206 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:18.593966 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:18.593971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:18.593985 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:18.594262 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:18.594559 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:18.594562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:18.594565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:18.594729 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:18.595107 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:18.595114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:18.595343 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 06:25:18.595566 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 06:25:18.595904 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":164,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:18.598101 9972 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38069
I20260810 06:25:18.598340 9972 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:18.598878 9972 heartbeater.cc:511] Master 127.4.73.62:38069 requested a full tablet report, sending...
I20260810 06:25:18.600530 9657 ts_manager.cc:194] Registered new tserver with Master: 63c21c61597c4296bfbcf79196375ec9 (127.4.73.1:36345)
I20260810 06:25:18.601632 9657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:53133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:18.613204 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38069
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:18.722383 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:18.722630 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:18.722669 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:18.726634 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:18.726686 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:18.726740 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:18.731869 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:18.732803 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:18.733685 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:18.743243 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:18.743252 9989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:18.743423 9981 server_base.cc:1034] running on GCE node
W20260810 06:25:18.743257 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:18.743970 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:18.744678 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:18.745839 9981 hybrid_clock.cc:648] HybridClock initialized: now 1786343118745815 us; error 39 us; skew 500 ppm
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:18.748700 9981 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:18.749557 9981 webserver.cc:460] Webserver started at http://127.4.73.2:43059/ using document root <none> and password file <none>
I20260810 06:25:18.750093 9981 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:18.750154 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:18.750382 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:18.752281 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "e0c40945372244dea40bb1a504e78256"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "b3601d70b258d30d2596dc061f2937cb"
server_key_iv: "6ad5da1621baae709727c3dae1318425"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.752910 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "e0c40945372244dea40bb1a504e78256"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "b3601d70b258d30d2596dc061f2937cb"
server_key_iv: "6ad5da1621baae709727c3dae1318425"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.756433 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:18.758738 10003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.759896 9981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:18.760037 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "e0c40945372244dea40bb1a504e78256"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "b3601d70b258d30d2596dc061f2937cb"
server_key_iv: "6ad5da1621baae709727c3dae1318425"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.760147 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:18.776175 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:18.779578 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:18.779793 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:18.780463 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:18.781349 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:18.781397 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.781468 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:18.781502 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.790773 9981 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:37829
I20260810 06:25:18.790817 10203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:37829 every 8 connection(s)
I20260810 06:25:18.791658 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:18.799387 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9981
I20260810 06:25:18.799532 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:18.800105 4388 external_mini_cluster.cc:1428] Setting key 994a375a9872f9270fbcf62c35031de1
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:18.805421 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:18.793401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:34127 (local address 127.4.73.62:38069)
0810 06:25:18.793531 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:18.793535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:18.794288 (+ 753us) server_negotiation.cc:407] Connection header received
0810 06:25:18.795740 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:18.795744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:18.795805 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:18.795923 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:18.797141 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.797660 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.798811 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.799030 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.799145 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:18.801690 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:18.801693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:18.801695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:18.801716 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:18.803102 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:18.803757 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:18.803762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:18.803766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:18.803873 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:18.804203 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:18.804209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:18.804212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:18.804475 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:18.804573 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:18.805039 (+ 466us) server_negotiation.cc:299] Negotiation successful
0810 06:25:18.805172 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:25:18.806016 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:18.793682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38069 (local address 127.4.73.2:34127)
0810 06:25:18.794141 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:18.794182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:18.795516 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:18.796098 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:18.796105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:18.796521 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:18.796993 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:18.797007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.797815 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.797819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:18.798677 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:18.798686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:18.799191 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:18.799195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:18.799361 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:18.799949 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:18.799965 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:18.801497 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:18.803220 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:18.803229 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:18.803246 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:18.803630 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:18.803967 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:18.803970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:18.803971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:18.804109 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:18.804587 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:18.804592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:18.804808 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 06:25:18.804954 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 06:25:18.805184 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":149,"threads_started":1}
I20260810 06:25:18.807245 10205 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38069
I20260810 06:25:18.807528 10205 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:18.807989 10205 heartbeater.cc:511] Master 127.4.73.62:38069 requested a full tablet report, sending...
I20260810 06:25:18.809065 9657 ts_manager.cc:194] Registered new tserver with Master: e0c40945372244dea40bb1a504e78256 (127.4.73.2:37829)
I20260810 06:25:18.809787 9657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:34127
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:18.826871 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38069
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:45825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:18.936314 10211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:18.936623 10211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:18.936700 10211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:18.941072 10211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:18.941121 10211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:18.941218 10211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:18.946142 10211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:45825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:18.947055 10211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:18.947960 10211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:18.957585 10228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:18.957679 10211 server_base.cc:1034] running on GCE node
W20260810 06:25:18.957618 10224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:18.957592 10225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:18.958268 10211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:18.958861 10211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:18.960047 10211 hybrid_clock.cc:648] HybridClock initialized: now 1786343118960029 us; error 28 us; skew 500 ppm
Aug 10 06:25:18 dist-test-slave-45dx krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:18.962975 10211 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:18.963837 10211 webserver.cc:460] Webserver started at http://127.4.73.3:40417/ using document root <none> and password file <none>
I20260810 06:25:18.964486 10211 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:18.964578 10211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:18.964803 10211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:18.966737 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "428e5067dd344bbca4b16c56560da40a"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "e39b7cda3037748ca13fd141ddaf44a6"
server_key_iv: "cd8c6a49a3bd8e1aadc612a09d323e6c"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.967298 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "428e5067dd344bbca4b16c56560da40a"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "e39b7cda3037748ca13fd141ddaf44a6"
server_key_iv: "cd8c6a49a3bd8e1aadc612a09d323e6c"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.970762 10211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:25:18.973052 10241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.974191 10211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:18.974325 10211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "428e5067dd344bbca4b16c56560da40a"
format_stamp: "Formatted at 2026-08-10 06:25:18 on dist-test-slave-45dx"
server_key: "e39b7cda3037748ca13fd141ddaf44a6"
server_key_iv: "cd8c6a49a3bd8e1aadc612a09d323e6c"
server_key_version: "encryptionkey@0"
I20260810 06:25:18.974431 10211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:18.990291 10211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:18.993444 10211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:18.993644 10211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:18.994203 10211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:18.995016 10211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:18.995087 10211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:18.995154 10211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:18.995196 10211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:19.003783 10211 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:42119
I20260810 06:25:19.003811 10437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:42119 every 8 connection(s)
I20260810 06:25:19.004602 10211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:19.013087 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10211
I20260810 06:25:19.013171 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:19.013511 4388 external_mini_cluster.cc:1428] Setting key c9b156f01a1d5ea68b15fb6bf7856e8c
Aug 10 06:25:19 dist-test-slave-45dx krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343118, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:19.018170 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:19.006355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:49855 (local address 127.4.73.62:38069)
0810 06:25:19.006495 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:19.006498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:19.007285 (+ 787us) server_negotiation.cc:407] Connection header received
0810 06:25:19.008620 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:19.008623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:19.008678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:19.008800 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:19.009892 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:19.010416 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:19.011559 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:19.011784 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:19.011896 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:19.014513 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:19.014516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:19.014518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:19.014540 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:19.016103 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:19.016686 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:19.016690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:19.016692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:19.016776 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:19.017134 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:19.017137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:19.017139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:19.017314 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:19.017429 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:19.017889 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 06:25:19.018007 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:25:19.018891 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:19.006696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38069 (local address 127.4.73.3:49855)
0810 06:25:19.007136 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:19.007176 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:19.008409 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:19.008965 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:19.008971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:19.009298 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:19.009733 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:19.009746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:19.010555 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:19.010561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:19.011422 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:19.011431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:19.011937 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:19.011941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:19.012111 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:19.012713 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:19.012729 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:19.014324 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:19.016225 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:19.016231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:19.016243 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:19.016578 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:19.016877 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:19.016881 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:19.016882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:19.017033 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:19.017429 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:19.017436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:19.017660 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 06:25:19.017831 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 06:25:19.018141 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":122,"threads_started":1}
I20260810 06:25:19.020002 10438 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38069
I20260810 06:25:19.020229 10438 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:19.020713 10438 heartbeater.cc:511] Master 127.4.73.62:38069 requested a full tablet report, sending...
I20260810 06:25:19.021704 9657 ts_manager.cc:194] Registered new tserver with Master: 428e5067dd344bbca4b16c56560da40a (127.4.73.3:42119)
I20260810 06:25:19.022370 9657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:49855
I20260810 06:25:19.027230 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:25:19 dist-test-slave-45dx krb5kdc[9572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343119, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 06:25:19 dist-test-slave-45dx krb5kdc[9572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343119, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:19.054515 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:19.045800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.4.73.62:38069)
0810 06:25:19.045985 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:19.045990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:19.046134 (+ 144us) server_negotiation.cc:407] Connection header received
0810 06:25:19.046303 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:19.046307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:19.046352 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:19.046444 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:19.047181 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:19.047700 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:19.048655 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:19.048869 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:19.048974 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:19.051404 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:19.051407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:19.051409 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:19.051431 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:19.052854 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:19.053284 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:19.053288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:19.053289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:19.053364 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:19.053639 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:19.053642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:19.053644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:19.053853 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:19.053999 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:19.054232 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 06:25:19.054344 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 06:25:19.054891 9657 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36016
I20260810 06:25:19.055721 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9752
I20260810 06:25:19.069940 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9981
I20260810 06:25:19.081773 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10211
I20260810 06:25:19.093674 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9594
2026-08-10T06:25:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2354 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:19 dist-test-slave-45dx krb5kdc[10459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:19 dist-test-slave-45dx krb5kdc[10459](info): set up 2 sockets
Aug 10 06:25:19 dist-test-slave-45dx krb5kdc[10459](info): commencing operation
krb5kdc: starting...
W20260810 06:25:20.360222 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:20 dist-test-slave-45dx krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:20.489495 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34441
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:20.600903 10478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:20.601171 10478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:20.601248 10478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:20.605124 10478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:20.605207 10478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:20.605242 10478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:20.605278 10478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:20.605306 10478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:20.610335 10478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:20.611300 10478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:20.612315 10478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:20.620517 10488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:20.620489 10485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:20.620555 10478 server_base.cc:1034] running on GCE node
W20260810 06:25:20.620508 10483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:20.621320 10478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:20.622320 10478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:20.623526 10478 hybrid_clock.cc:648] HybridClock initialized: now 1786343120623509 us; error 30 us; skew 500 ppm
Aug 10 06:25:20 dist-test-slave-45dx krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343120, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:20.626435 10478 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:20.627296 10478 webserver.cc:460] Webserver started at http://127.4.73.62:44841/ using document root <none> and password file <none>
I20260810 06:25:20.627854 10478 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:20.627952 10478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:20.628230 10478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:20.630244 10478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "6a08e000fd924363bbaf0928a6f38cec"
format_stamp: "Formatted at 2026-08-10 06:25:20 on dist-test-slave-45dx"
server_key: "c9fe8411bf6ae0c3cf453abc3dc9df4a"
server_key_iv: "010b39e52f16f63cc14305c587524162"
server_key_version: "encryptionkey@0"
I20260810 06:25:20.630815 10478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "6a08e000fd924363bbaf0928a6f38cec"
format_stamp: "Formatted at 2026-08-10 06:25:20 on dist-test-slave-45dx"
server_key: "c9fe8411bf6ae0c3cf453abc3dc9df4a"
server_key_iv: "010b39e52f16f63cc14305c587524162"
server_key_version: "encryptionkey@0"
I20260810 06:25:20.634407 10478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 06:25:20.636685 10497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:20.637794 10478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:20.637944 10478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "6a08e000fd924363bbaf0928a6f38cec"
format_stamp: "Formatted at 2026-08-10 06:25:20 on dist-test-slave-45dx"
server_key: "c9fe8411bf6ae0c3cf453abc3dc9df4a"
server_key_iv: "010b39e52f16f63cc14305c587524162"
server_key_version: "encryptionkey@0"
I20260810 06:25:20.638055 10478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:20.664729 10478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:20.668149 10478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:20.668416 10478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:20.675472 10478 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:34441
I20260810 06:25:20.675500 10581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:34441 every 8 connection(s)
I20260810 06:25:20.676499 10478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:20.676791 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10478
I20260810 06:25:20.676952 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:20.677307 4388 external_mini_cluster.cc:1428] Setting key e3d4ae3b9540cae9e56f109617e3f560
I20260810 06:25:20.679620 10582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:20.685168 10582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec: Bootstrap starting.
Aug 10 06:25:20 dist-test-slave-45dx krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:20.687327 10582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:20.688014 10582 log.cc:824] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:20.690137 10582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec: No bootstrap required, opened a new log
I20260810 06:25:20.691166 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:20.678709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.4.73.62:34441)
0810 06:25:20.679134 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:20.679148 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:20.679181 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:25:20.679841 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:20.679858 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:20.680164 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:20.680544 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:20.681345 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:20.682340 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:20.683215 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:20.683670 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:20.683837 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:20.686340 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:20.686345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:20.686357 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:20.686385 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:20.688111 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:20.688673 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:20.688678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:20.688682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:20.688875 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:20.689171 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:20.689174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:20.689176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:20.689648 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:20.689868 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:20.690016 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 06:25:20.690295 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":140,"threads_started":1}
I20260810 06:25:20.693776 10582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a08e000fd924363bbaf0928a6f38cec" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34441 } }
I20260810 06:25:20.694043 10582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:20.694084 10582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a08e000fd924363bbaf0928a6f38cec, State: Initialized, Role: FOLLOWER
I20260810 06:25:20.694495 10582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a08e000fd924363bbaf0928a6f38cec" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34441 } }
I20260810 06:25:20.694625 10582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:20.694671 10582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:20.694741 10582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:20.695518 10582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a08e000fd924363bbaf0928a6f38cec" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34441 } }
I20260810 06:25:20.695786 10582 leader_election.cc:302] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a08e000fd924363bbaf0928a6f38cec; no voters:
I20260810 06:25:20.696045 10582 leader_election.cc:288] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:20.696190 10590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:20.696475 10590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [term 1 LEADER]: Becoming Leader. State: Replica: 6a08e000fd924363bbaf0928a6f38cec, State: Running, Role: LEADER
I20260810 06:25:20.696828 10590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a08e000fd924363bbaf0928a6f38cec" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34441 } }
I20260810 06:25:20.697230 10582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:20.698457 10592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a08e000fd924363bbaf0928a6f38cec. Latest consensus state: current_term: 1 leader_uuid: "6a08e000fd924363bbaf0928a6f38cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a08e000fd924363bbaf0928a6f38cec" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34441 } } }
I20260810 06:25:20.698591 10592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:20.699005 10601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:20.698993 10591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a08e000fd924363bbaf0928a6f38cec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a08e000fd924363bbaf0928a6f38cec" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34441 } } }
I20260810 06:25:20.699970 10591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:20.702389 10601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:20.707521 10601 catalog_manager.cc:1347] Generated new cluster ID: 73301df7071b4d46af86a777eae5c28b
I20260810 06:25:20.707594 10601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:20.722268 10601 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:20.723171 10601 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:20.730150 10601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a08e000fd924363bbaf0928a6f38cec: Generated new TSK 0
I20260810 06:25:20.730763 10601 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:20.765007 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34441
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:20.875301 10622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:20.875532 10622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:20.875571 10622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:20.879540 10622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:20.879590 10622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:20.879644 10622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:20.884613 10622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:20.885486 10622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:20.886381 10622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:20.895421 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:20.895431 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:20.895467 10622 server_base.cc:1034] running on GCE node
W20260810 06:25:20.895421 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:20.896116 10622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:20.896727 10622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:20.897900 10622 hybrid_clock.cc:648] HybridClock initialized: now 1786343120897882 us; error 28 us; skew 500 ppm
Aug 10 06:25:20 dist-test-slave-45dx krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343120, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:20.900821 10622 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:20.901603 10622 webserver.cc:460] Webserver started at http://127.4.73.1:38221/ using document root <none> and password file <none>
I20260810 06:25:20.902168 10622 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:20.902258 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:20.902483 10622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:20.904469 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "3f4ed60e8bba40a8bd3e1aa9db5be9fd"
format_stamp: "Formatted at 2026-08-10 06:25:20 on dist-test-slave-45dx"
server_key: "69e953514ab59e00e495e815da365678"
server_key_iv: "50663fa051d57fb6aa1a31f0ac1e79ac"
server_key_version: "encryptionkey@0"
I20260810 06:25:20.905045 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "3f4ed60e8bba40a8bd3e1aa9db5be9fd"
format_stamp: "Formatted at 2026-08-10 06:25:20 on dist-test-slave-45dx"
server_key: "69e953514ab59e00e495e815da365678"
server_key_iv: "50663fa051d57fb6aa1a31f0ac1e79ac"
server_key_version: "encryptionkey@0"
I20260810 06:25:20.908560 10622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 06:25:20.910820 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:20.911969 10622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:20.912096 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "3f4ed60e8bba40a8bd3e1aa9db5be9fd"
format_stamp: "Formatted at 2026-08-10 06:25:20 on dist-test-slave-45dx"
server_key: "69e953514ab59e00e495e815da365678"
server_key_iv: "50663fa051d57fb6aa1a31f0ac1e79ac"
server_key_version: "encryptionkey@0"
I20260810 06:25:20.912220 10622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:20.925508 10622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:20.928249 10622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:20.928480 10622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:20.929031 10622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:20.929886 10622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:20.929960 10622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:20.930033 10622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:20.930075 10622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:20.938633 10622 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:43919
I20260810 06:25:20.938675 10826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:43919 every 8 connection(s)
I20260810 06:25:20.939440 10622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:20.940858 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10622
I20260810 06:25:20.940986 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:20.941329 4388 external_mini_cluster.cc:1428] Setting key 43c3797b609fb42acebfc23ff01c7c52
Aug 10 06:25:20 dist-test-slave-45dx krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343120, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:25:20.953534 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:20.941271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:53229 (local address 127.4.73.62:34441)
0810 06:25:20.941430 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:20.941436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:20.942540 (+ 1104us) server_negotiation.cc:407] Connection header received
0810 06:25:20.943914 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:20.943918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:20.943969 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:20.944058 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:20.945193 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:20.945705 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:20.946876 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:20.947131 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:20.947250 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:20.950077 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:20.950081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:20.950083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:20.950106 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:20.951632 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:20.952102 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:20.952104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:20.952106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:20.952194 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:20.952624 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:20.952626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:20.952627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:20.952795 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:20.952873 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:20.953320 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 06:25:20.953418 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:20.954344 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:20.941802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34441 (local address 127.4.73.1:53229)
0810 06:25:20.942356 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:20.942436 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:20.943707 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:20.944223 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:20.944230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:20.944616 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:20.945047 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:20.945061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:20.945832 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:20.945836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:20.946733 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:20.946743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:20.947300 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:20.947304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:20.947466 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:20.948038 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:20.948053 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:20.949884 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:20.951741 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:20.951745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:20.951755 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:20.952013 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:20.952292 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:20.952297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:20.952301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:20.952534 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:20.952888 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:20.952892 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:20.953103 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 06:25:20.953256 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 06:25:20.953494 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":158,"threads_started":1}
I20260810 06:25:20.955479 10827 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34441
I20260810 06:25:20.955729 10827 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:20.956279 10827 heartbeater.cc:511] Master 127.4.73.62:34441 requested a full tablet report, sending...
I20260810 06:25:20.957969 10527 ts_manager.cc:194] Registered new tserver with Master: 3f4ed60e8bba40a8bd3e1aa9db5be9fd (127.4.73.1:43919)
I20260810 06:25:20.959627 10527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:53229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:20.968942 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34441
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:21.080242 10835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:21.080513 10835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:21.080552 10835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:21.084687 10835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:21.084743 10835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:21.084797 10835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:21.089800 10835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:21.090768 10835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:21.091759 10835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:21.101625 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:21.101615 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:21.101642 10848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:21.101986 10835 server_base.cc:1034] running on GCE node
I20260810 06:25:21.102377 10835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:21.103024 10835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:21.104226 10835 hybrid_clock.cc:648] HybridClock initialized: now 1786343121104215 us; error 26 us; skew 500 ppm
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343121, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:21.107079 10835 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:21.107888 10835 webserver.cc:460] Webserver started at http://127.4.73.2:34425/ using document root <none> and password file <none>
I20260810 06:25:21.108475 10835 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:21.108567 10835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:21.108803 10835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:21.110826 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "eafabafbf36743ed8be61fe564fc274d"
format_stamp: "Formatted at 2026-08-10 06:25:21 on dist-test-slave-45dx"
server_key: "a365e434cec83504cbd4f9cd77f55868"
server_key_iv: "67b9213383b1a452b1e628d6217105bd"
server_key_version: "encryptionkey@0"
I20260810 06:25:21.111418 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "eafabafbf36743ed8be61fe564fc274d"
format_stamp: "Formatted at 2026-08-10 06:25:21 on dist-test-slave-45dx"
server_key: "a365e434cec83504cbd4f9cd77f55868"
server_key_iv: "67b9213383b1a452b1e628d6217105bd"
server_key_version: "encryptionkey@0"
I20260810 06:25:21.115198 10835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 06:25:21.117703 10857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:21.118937 10835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:21.119076 10835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "eafabafbf36743ed8be61fe564fc274d"
format_stamp: "Formatted at 2026-08-10 06:25:21 on dist-test-slave-45dx"
server_key: "a365e434cec83504cbd4f9cd77f55868"
server_key_iv: "67b9213383b1a452b1e628d6217105bd"
server_key_version: "encryptionkey@0"
I20260810 06:25:21.119189 10835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:21.130820 10835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:21.134032 10835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:21.134227 10835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:21.134871 10835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:21.135650 10835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:21.135722 10835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:21.135792 10835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:21.135834 10835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:21.144399 11054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:42399 every 8 connection(s)
I20260810 06:25:21.144407 10835 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:42399
I20260810 06:25:21.145279 10835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343121, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:21.154896 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10835
I20260810 06:25:21.155030 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:21.155325 4388 external_mini_cluster.cc:1428] Setting key 894fce1ee4e21f2ee1fed3e75ddf7242
I20260810 06:25:21.158671 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:21.146949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:47357 (local address 127.4.73.62:34441)
0810 06:25:21.147080 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:21.147084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:21.147824 (+ 740us) server_negotiation.cc:407] Connection header received
0810 06:25:21.149230 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:21.149233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:21.149281 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:21.149352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:21.150406 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.150984 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.152073 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.152278 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.152416 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:21.154610 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:21.154613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:21.154615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:21.154642 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:21.156469 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.157090 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.157093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.157095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.157176 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.157591 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.157594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.157595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.157776 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:21.157851 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:21.158458 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 06:25:21.158557 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:25:21.159642 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:21.147238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34441 (local address 127.4.73.2:47357)
0810 06:25:21.147671 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:21.147711 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:21.149038 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:21.149522 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:21.149528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:21.149840 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:21.150282 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:21.150294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.151092 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.151095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:21.151959 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:21.151968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.152437 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.152440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:21.152578 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:21.153057 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:21.153072 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:21.154458 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:21.156606 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:21.156614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:21.156630 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:21.156988 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:21.157277 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:21.157281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:21.157283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:21.157483 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:21.157889 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:21.157897 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:21.158208 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 06:25:21.158417 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 06:25:21.158763 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
I20260810 06:25:21.160691 11055 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34441
I20260810 06:25:21.160934 11055 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:21.161384 11055 heartbeater.cc:511] Master 127.4.73.62:34441 requested a full tablet report, sending...
I20260810 06:25:21.162425 10527 ts_manager.cc:194] Registered new tserver with Master: eafabafbf36743ed8be61fe564fc274d (127.4.73.2:42399)
I20260810 06:25:21.162992 10527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:47357
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:21.183249 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34441
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:21.297968 11062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:21.298199 11062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:21.298237 11062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:21.302279 11062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:21.302331 11062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:21.302387 11062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:21.307420 11062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:21.308372 11062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:21.309299 11062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:21.318641 11075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:21.318794 11062 server_base.cc:1034] running on GCE node
W20260810 06:25:21.318673 11070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:21.318615 11071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:21.319357 11062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:21.319975 11062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:21.321174 11062 hybrid_clock.cc:648] HybridClock initialized: now 1786343121321164 us; error 27 us; skew 500 ppm
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343121, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:21.323992 11062 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:21.324864 11062 webserver.cc:460] Webserver started at http://127.4.73.3:39707/ using document root <none> and password file <none>
I20260810 06:25:21.325410 11062 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:21.325506 11062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:21.325735 11062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:21.327773 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "1f57c0153a5d4bb39c931069a118a43e"
format_stamp: "Formatted at 2026-08-10 06:25:21 on dist-test-slave-45dx"
server_key: "c828359653e74779da416aa6b30d56a9"
server_key_iv: "754aa381d12d662e6858fac78b959cb2"
server_key_version: "encryptionkey@0"
I20260810 06:25:21.328531 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "1f57c0153a5d4bb39c931069a118a43e"
format_stamp: "Formatted at 2026-08-10 06:25:21 on dist-test-slave-45dx"
server_key: "c828359653e74779da416aa6b30d56a9"
server_key_iv: "754aa381d12d662e6858fac78b959cb2"
server_key_version: "encryptionkey@0"
I20260810 06:25:21.332661 11062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:25:21.335409 11084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:21.336839 11062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:21.336995 11062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "1f57c0153a5d4bb39c931069a118a43e"
format_stamp: "Formatted at 2026-08-10 06:25:21 on dist-test-slave-45dx"
server_key: "c828359653e74779da416aa6b30d56a9"
server_key_iv: "754aa381d12d662e6858fac78b959cb2"
server_key_version: "encryptionkey@0"
I20260810 06:25:21.337126 11062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:21.352824 11062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:21.355618 11062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:21.355829 11062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:21.356498 11062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:21.357332 11062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:21.357402 11062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:21.357468 11062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:21.357515 11062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:21.366345 11062 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:37951
I20260810 06:25:21.366403 11270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:37951 every 8 connection(s)
I20260810 06:25:21.367201 11062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:21.369840 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 11062
I20260810 06:25:21.369958 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:21.370297 4388 external_mini_cluster.cc:1428] Setting key e2021fbc79cd6d53f06b408c99277c83
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343121, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:21.381031 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:21.369035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:33259 (local address 127.4.73.62:34441)
0810 06:25:21.369175 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:21.369179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:21.370105 (+ 926us) server_negotiation.cc:407] Connection header received
0810 06:25:21.372013 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:21.372016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:21.372069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:21.372162 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:21.373428 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.373938 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.375035 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.375249 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.375336 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:21.377677 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:21.377680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:21.377682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:21.377706 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:21.379185 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.379647 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.379650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.379651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.379726 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.380040 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.380042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.380043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.380219 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:21.380293 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:21.380807 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 06:25:21.380905 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:25:21.381817 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:21.369387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34441 (local address 127.4.73.3:33259)
0810 06:25:21.369916 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:21.369996 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:21.371770 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:21.372360 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:21.372368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:21.372826 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:21.373283 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:21.373296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.374045 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.374047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:21.374906 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:21.374915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.375386 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.375389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:21.375522 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:21.376057 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:21.376072 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:21.377527 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:21.379290 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:21.379294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:21.379309 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:21.379550 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:21.379805 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:21.379808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:21.379810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:21.379934 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:21.380347 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:21.380352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:21.380598 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 06:25:21.380749 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 06:25:21.380996 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260810 06:25:21.383054 11278 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34441
I20260810 06:25:21.383302 11278 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:21.383888 11278 heartbeater.cc:511] Master 127.4.73.62:34441 requested a full tablet report, sending...
I20260810 06:25:21.384968 10527 ts_manager.cc:194] Registered new tserver with Master: 1f57c0153a5d4bb39c931069a118a43e (127.4.73.3:37951)
I20260810 06:25:21.385506 10527 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:33259
I20260810 06:25:21.396076 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[10459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:21.425474 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:21.415205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.4.73.1:43919)
0810 06:25:21.415555 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:21.415574 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:21.415607 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:25:21.415663 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:21.415669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:21.415935 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:21.416076 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:21.416916 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.417793 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.418679 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.419078 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.419215 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:21.421619 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:21.421628 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:21.421641 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:21.421666 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:21.423556 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.424042 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.424047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.424048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.424142 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.424467 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.424470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.424472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.424723 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:21.424963 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:21.425138 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 06:25:21.425292 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":151,"threads_started":1}
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[10459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:21.435738 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:21.428290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.4.73.62:34441)
0810 06:25:21.428432 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:21.428438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:21.428496 (+ 58us) server_negotiation.cc:407] Connection header received
0810 06:25:21.428644 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:21.428647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:21.428701 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:21.428792 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:21.429361 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.429861 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.430622 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:21.430841 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:21.430958 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:21.433267 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:21.433270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:21.433272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:21.433296 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:21.434469 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.434811 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.434814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.434815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.434886 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:21.435094 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:21.435096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:21.435097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:21.435281 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:21.435377 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:21.435549 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 06:25:21.435633 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 06:25:21.436226 10527 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43788
I20260810 06:25:21.436853 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10622
I20260810 06:25:21.450106 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10835
I20260810 06:25:21.467557 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 11062
I20260810 06:25:21.479732 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10478
2026-08-10T06:25:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2387 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[11299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[11299](info): set up 2 sockets
Aug 10 06:25:21 dist-test-slave-45dx krb5kdc[11299](info): commencing operation
krb5kdc: starting...
W20260810 06:25:22.857751 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 06:25:23.090171 4388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.232s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 06:25:23.243124 4388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.153s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 06:25:23.460170 4388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.217s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/test-user.keytab.
W20260810 06:25:24.017663 4388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.557s user 0.002s sys 0.000s
Aug 10 06:25:24 dist-test-slave-45dx krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
W20260810 06:25:24.267510 4388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.4.73.62: real 0.224s user 0.000s sys 0.003s
I20260810 06:25:24.268270 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34719
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:24.381654 11320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:24.381928 11320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:24.381969 11320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:24.385840 11320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:24.385886 11320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:24.385905 11320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:24.385921 11320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:24.385936 11320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:24.390892 11320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:24.391799 11320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:24.392756 11320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:25:24.400866 11320 server_base.cc:1034] running on GCE node
W20260810 06:25:24.400821 11327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:24.400810 11330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:24.400835 11328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:24.401587 11320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:24.402634 11320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:24.403829 11320 hybrid_clock.cc:648] HybridClock initialized: now 1786343124403810 us; error 30 us; skew 500 ppm
Aug 10 06:25:24 dist-test-slave-45dx krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343124, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:24.413101 11320 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:24.413808 11320 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:24.413923 11320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:24.414162 11320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:24.416157 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "4a23a78f48e64af990a55b5bc077741a"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "239eb452478727434e9fb8c8530f99ff"
server_key_iv: "804c222dd8f25c6e554068966ff19b9c"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.416781 11320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "4a23a78f48e64af990a55b5bc077741a"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "239eb452478727434e9fb8c8530f99ff"
server_key_iv: "804c222dd8f25c6e554068966ff19b9c"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.420459 11320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:25:24.422775 11346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:24.423964 11320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:24.424109 11320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "4a23a78f48e64af990a55b5bc077741a"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "239eb452478727434e9fb8c8530f99ff"
server_key_iv: "804c222dd8f25c6e554068966ff19b9c"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.424217 11320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:24.456418 11320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:24.459170 11320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:24.459369 11320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:24.466141 11320 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:34719
I20260810 06:25:24.466434 11425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:34719 every 8 connection(s)
I20260810 06:25:24.466821 11320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:24.469434 11426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:24.474742 11426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a: Bootstrap starting.
I20260810 06:25:24.477003 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 11320
I20260810 06:25:24.477147 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:24.477281 11426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:24.477494 4388 external_mini_cluster.cc:1428] Setting key 09b49e786dad0d6964b592e27925b3d5
I20260810 06:25:24.478214 11426 log.cc:824] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:24.480443 11426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a: No bootstrap required, opened a new log
I20260810 06:25:24.484863 11426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a23a78f48e64af990a55b5bc077741a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34719 } }
I20260810 06:25:24.485172 11426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:24.485253 11426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a23a78f48e64af990a55b5bc077741a, State: Initialized, Role: FOLLOWER
I20260810 06:25:24.485746 11426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a23a78f48e64af990a55b5bc077741a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34719 } }
I20260810 06:25:24.485880 11426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:24.485963 11426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:24.486060 11426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 0 FOLLOWER]: Advancing to term 1
Aug 10 06:25:24 dist-test-slave-45dx krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:24.487213 11426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a23a78f48e64af990a55b5bc077741a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34719 } }
I20260810 06:25:24.487609 11426 leader_election.cc:302] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a23a78f48e64af990a55b5bc077741a; no voters:
I20260810 06:25:24.487929 11426 leader_election.cc:288] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:24.488065 11435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:24.488317 11435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [term 1 LEADER]: Becoming Leader. State: Replica: 4a23a78f48e64af990a55b5bc077741a, State: Running, Role: LEADER
I20260810 06:25:24.488653 11435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a23a78f48e64af990a55b5bc077741a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34719 } }
I20260810 06:25:24.489310 11426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:24.490295 11436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a23a78f48e64af990a55b5bc077741a. Latest consensus state: current_term: 1 leader_uuid: "4a23a78f48e64af990a55b5bc077741a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a23a78f48e64af990a55b5bc077741a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34719 } } }
I20260810 06:25:24.490428 11436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:24.490723 11438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a23a78f48e64af990a55b5bc077741a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a23a78f48e64af990a55b5bc077741a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34719 } } }
I20260810 06:25:24.490837 11438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:24.491168 11453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:24.493582 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:24.478574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.4.73.62:34719)
0810 06:25:24.479022 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:24.479039 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:24.479082 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:25:24.479879 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:24.479900 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:24.480289 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:24.480661 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:24.481317 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.482770 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.483638 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.484572 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.484763 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:24.487610 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:24.487615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:24.487627 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:24.487651 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:24.490083 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:24.490669 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:24.490675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:24.490681 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:24.490837 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:24.491153 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:24.491159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:24.491164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:24.491700 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:24.492277 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:24.492441 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 06:25:24.492718 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260810 06:25:24.495328 11453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:24.501091 11453 catalog_manager.cc:1347] Generated new cluster ID: dc3e1f6253964aedb53ea55d96384994
I20260810 06:25:24.501150 11453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:24.520964 11453 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:24.521847 11453 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:24.529793 11453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a: Generated new TSK 0
I20260810 06:25:24.530562 11453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:24.565265 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34719
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:24.674358 11468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:24.674641 11468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:24.674721 11468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:24.678679 11468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:24.678766 11468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:24.678861 11468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:24.683987 11468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:24.684994 11468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:24.686504 11468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:24.695968 11482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:24.695968 11480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:24.696103 11478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:24.696123 11468 server_base.cc:1034] running on GCE node
I20260810 06:25:24.696707 11468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:24.697291 11468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:24.698448 11468 hybrid_clock.cc:648] HybridClock initialized: now 1786343124698417 us; error 44 us; skew 500 ppm
Aug 10 06:25:24 dist-test-slave-45dx krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343124, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:24.701367 11468 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:24.702111 11468 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:24.702178 11468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:24.702386 11468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:24.704294 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "09152cbf4f6a4ab1a01223fdbf831a5d"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "ca45d60ed6c7446b95853b51f8fa1e6f"
server_key_iv: "8f9ca18c761c7767c7ae34d08cfa9f22"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.704936 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "09152cbf4f6a4ab1a01223fdbf831a5d"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "ca45d60ed6c7446b95853b51f8fa1e6f"
server_key_iv: "8f9ca18c761c7767c7ae34d08cfa9f22"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.708612 11468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:25:24.711035 11491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:24.712286 11468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:24.712488 11468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "09152cbf4f6a4ab1a01223fdbf831a5d"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "ca45d60ed6c7446b95853b51f8fa1e6f"
server_key_iv: "8f9ca18c761c7767c7ae34d08cfa9f22"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.712607 11468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:24.745229 11468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:24.748771 11468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:24.748991 11468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:24.749598 11468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:24.750445 11468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:24.750519 11468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:24.750593 11468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:24.750648 11468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:24.759871 11468 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:38423
I20260810 06:25:24.759904 11678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:38423 every 8 connection(s)
I20260810 06:25:24.760548 11468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:24.761794 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 11468
I20260810 06:25:24.761919 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:24.762202 4388 external_mini_cluster.cc:1428] Setting key e06ffc24fced6e41bfaf117bd2d03445
Aug 10 06:25:24 dist-test-slave-45dx krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343124, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:25:24.775568 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:24.762394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:34661 (local address 127.4.73.62:34719)
0810 06:25:24.762567 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:24.762572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:24.763536 (+ 964us) server_negotiation.cc:407] Connection header received
0810 06:25:24.765208 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:24.765215 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:24.765278 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:24.765401 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:24.766996 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.767528 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.768820 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.769073 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.769197 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:24.771676 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:24.771682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:24.771685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:24.771720 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:24.773412 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:24.773957 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:24.773961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:24.773965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:24.774065 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:24.774418 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:24.774423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:24.774425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:24.774628 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:24.774765 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:24.775253 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 06:25:24.775388 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:24.776437 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:24.762841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34719 (local address 127.4.73.1:34661)
0810 06:25:24.763364 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:24.763416 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:24.764943 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:24.765581 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:24.765587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:24.766120 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:24.766826 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:24.766847 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.767653 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.767656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:24.768687 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:24.768698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.769234 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.769238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:24.769418 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:24.770031 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:24.770047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:24.771507 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:24.773542 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:24.773548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:24.773562 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:24.773821 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:24.774165 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:24.774170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:24.774172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:24.774317 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:24.774764 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:24.774770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:24.775016 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 06:25:24.775210 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 06:25:24.775541 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":165,"threads_started":1}
I20260810 06:25:24.777828 11682 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34719
I20260810 06:25:24.778080 11682 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:24.778549 11682 heartbeater.cc:511] Master 127.4.73.62:34719 requested a full tablet report, sending...
I20260810 06:25:24.780068 11373 ts_manager.cc:194] Registered new tserver with Master: 09152cbf4f6a4ab1a01223fdbf831a5d (127.4.73.1:38423)
I20260810 06:25:24.781481 11373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:34661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:24.791555 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34719
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:24.902220 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:24.902448 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:24.902490 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:24.906591 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:24.906654 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:24.906713 11691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:24.911981 11691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:24.912992 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:24.913887 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:24.923182 11703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:24.923208 11706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:24.923223 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:24.923429 11691 server_base.cc:1034] running on GCE node
I20260810 06:25:24.923874 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:24.924643 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:24.925810 11691 hybrid_clock.cc:648] HybridClock initialized: now 1786343124925779 us; error 38 us; skew 500 ppm
Aug 10 06:25:24 dist-test-slave-45dx krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343124, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:24.928843 11691 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:24.929544 11691 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:24.929610 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:24.929817 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:24.931759 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "a2573b4f200645a49c84fc5f5e0dcfd1"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "1229635f93b8b8d917a6391e35775ede"
server_key_iv: "cc85f9f7be050905771428a53e47ecd8"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.932364 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "a2573b4f200645a49c84fc5f5e0dcfd1"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "1229635f93b8b8d917a6391e35775ede"
server_key_iv: "cc85f9f7be050905771428a53e47ecd8"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.935913 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:25:24.938267 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:24.939429 11691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:24.939559 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "a2573b4f200645a49c84fc5f5e0dcfd1"
format_stamp: "Formatted at 2026-08-10 06:25:24 on dist-test-slave-45dx"
server_key: "1229635f93b8b8d917a6391e35775ede"
server_key_iv: "cc85f9f7be050905771428a53e47ecd8"
server_key_version: "encryptionkey@0"
I20260810 06:25:24.939659 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:24.952021 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:24.955258 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:24.955456 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:24.955996 11691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:24.956848 11691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:24.956897 11691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:24.956965 11691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:24.957002 11691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:24.965685 11691 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:37315
I20260810 06:25:24.965708 11897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:37315 every 8 connection(s)
I20260810 06:25:24.966214 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:24.967978 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 11691
I20260810 06:25:24.968148 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:24.968462 4388 external_mini_cluster.cc:1428] Setting key 38034975b99292f33d8c13341f5d74f4
Aug 10 06:25:24 dist-test-slave-45dx krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343124, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:25:24.981585 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:24.967905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:47685 (local address 127.4.73.62:34719)
0810 06:25:24.968064 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:24.968069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:24.969027 (+ 958us) server_negotiation.cc:407] Connection header received
0810 06:25:24.970689 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:24.970692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:24.970734 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:24.970801 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:24.972343 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.973175 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.974861 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.975128 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.975224 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:24.977993 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:24.977997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:24.977998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:24.978022 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:24.979493 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:24.979984 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:24.979988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:24.979993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:24.980078 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:24.980490 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:24.980493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:24.980495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:24.980679 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:24.980788 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:24.981349 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 06:25:24.981451 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:25:24.982326 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:24.968339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34719 (local address 127.4.73.2:47685)
0810 06:25:24.968878 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:24.968929 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:24.970478 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:24.970978 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:24.970990 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:24.971477 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:24.972142 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:24.972157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.973296 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.973298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:24.974727 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:24.974743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:24.975307 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:24.975315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:24.975524 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:24.976079 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:24.976105 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:24.977824 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:24.979606 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:24.979612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:24.979625 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:24.979875 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:24.980173 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:24.980176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:24.980178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:24.980373 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:24.980789 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:24.980796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:24.981032 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 06:25:24.981247 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 06:25:24.981501 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":139,"threads_started":1}
I20260810 06:25:24.983549 11900 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34719
I20260810 06:25:24.983778 11900 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:24.984215 11900 heartbeater.cc:511] Master 127.4.73.62:34719 requested a full tablet report, sending...
I20260810 06:25:24.985231 11373 ts_manager.cc:194] Registered new tserver with Master: a2573b4f200645a49c84fc5f5e0dcfd1 (127.4.73.2:37315)
I20260810 06:25:24.985832 11373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:47685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:24.997097 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34719
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:25.113794 11908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:25.114027 11908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:25.114068 11908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:25.117882 11908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:25.117924 11908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:25.117977 11908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:25.122921 11908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:25.123766 11908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:25.124653 11908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:25.133685 11915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:25.133687 11917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:25.133719 11908 server_base.cc:1034] running on GCE node
W20260810 06:25:25.133685 11920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:25.134392 11908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:25.134959 11908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:25.136121 11908 hybrid_clock.cc:648] HybridClock initialized: now 1786343125136102 us; error 28 us; skew 500 ppm
Aug 10 06:25:25 dist-test-slave-45dx krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343125, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:25.139047 11908 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:25.139793 11908 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:25.139863 11908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:25.140090 11908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:25.142169 11908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "b2e3de33d9e14c348e3532f6a8d7a207"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "3744086999edf0cb56847f5c6594a419"
server_key_iv: "9bd5ffee23ab579e77d146f3ed0fc174"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.142777 11908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "b2e3de33d9e14c348e3532f6a8d7a207"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "3744086999edf0cb56847f5c6594a419"
server_key_iv: "9bd5ffee23ab579e77d146f3ed0fc174"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.146445 11908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:25:25.148777 11932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:25.149884 11908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:25.150023 11908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "b2e3de33d9e14c348e3532f6a8d7a207"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "3744086999edf0cb56847f5c6594a419"
server_key_iv: "9bd5ffee23ab579e77d146f3ed0fc174"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.150143 11908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:25.178486 11908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:25.181516 11908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:25.181780 11908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:25.182415 11908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:25.183341 11908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:25.183420 11908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:25.183492 11908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:25.183533 11908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:25.192529 11908 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:41701
I20260810 06:25:25.192552 12127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:41701 every 8 connection(s)
I20260810 06:25:25.193091 11908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:25.193406 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 11908
I20260810 06:25:25.193532 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:25.193913 4388 external_mini_cluster.cc:1428] Setting key 1d6e2243b3c7dae17cae55764fbe8e33
Aug 10 06:25:25 dist-test-slave-45dx krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343125, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:25.207206 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.194962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:45219 (local address 127.4.73.62:34719)
0810 06:25:25.195089 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.195094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.196100 (+ 1006us) server_negotiation.cc:407] Connection header received
0810 06:25:25.197577 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.197581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.197654 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.197748 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:25.199029 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.199538 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.200789 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.201021 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.201136 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.203583 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:25.203586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:25.203588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:25.203611 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:25.205030 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:25.205612 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:25.205621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:25.205622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:25.205702 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:25.206054 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:25.206057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:25.206059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:25.206242 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:25.206355 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.206901 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.207037 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:25:25.207957 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.195438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34719 (local address 127.4.73.3:45219)
0810 06:25:25.195954 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:25.196013 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:25.197355 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:25.197951 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:25.197962 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:25.198380 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:25.198863 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.198878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.199668 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.199671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.200650 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.200661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.201174 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.201179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.201352 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.201968 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:25.201985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:25.203398 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:25.205151 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:25.205157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:25.205170 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:25.205500 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:25.205798 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:25.205801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:25.205803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:25.205948 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:25.206355 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:25.206362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:25.206632 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 06:25:25.206855 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 06:25:25.207164 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":100,"threads_started":1}
I20260810 06:25:25.209004 12128 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34719
I20260810 06:25:25.209241 12128 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:25.209679 12128 heartbeater.cc:511] Master 127.4.73.62:34719 requested a full tablet report, sending...
I20260810 06:25:25.210644 11373 ts_manager.cc:194] Registered new tserver with Master: b2e3de33d9e14c348e3532f6a8d7a207 (127.4.73.3:41701)
I20260810 06:25:25.211205 11373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:45219
I20260810 06:25:25.219843 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:25:25.229547 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.221861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.4.73.62:34719)
0810 06:25:25.221995 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.221998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.222185 (+ 187us) server_negotiation.cc:407] Connection header received
0810 06:25:25.222287 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.222289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.222328 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.222394 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:25.223094 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.223832 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.224889 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.225113 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.225191 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.226576 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:25.226579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:25.226581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:25.226599 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:25.228020 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:25.228511 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:25.228514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:25.228515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:25.228591 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:25.228859 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:25.228861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:25.228862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:25.229077 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:25.229180 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.229344 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.229433 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:25:25.232389 11373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:25.235261 11373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:25.250247 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.243909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:37315 (local address 127.0.0.1:51428)
0810 06:25:25.244654 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:25.244700 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:25.244816 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:25.245803 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:25.245806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:25.245828 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:25.245933 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.245939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.247642 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.247644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.248774 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.248784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.249771 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.249773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.249850 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.249878 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 06:25:25.249948 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 06:25:25.250087 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260810 06:25:25.250248 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.244061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.4.73.2:37315)
0810 06:25:25.245291 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.245302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.245324 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:25:25.245413 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.245419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.245644 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.245790 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:25.246038 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.247224 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.248899 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.249649 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.249779 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.249924 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.249993 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.250087 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":117,"threads_started":1}
I20260810 06:25:25.250744 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.244034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:38423 (local address 127.0.0.1:38570)
0810 06:25:25.244653 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:25.244699 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:25.244818 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:25.245709 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:25.245718 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:25.245747 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:25.245882 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.245889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.247378 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.247383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.248990 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.249002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.250041 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.250044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.250119 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.250139 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:25:25.250210 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:25:25.250286 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":141,"spinlock_wait_cycles":2432,"thread_start_us":73,"threads_started":1}
I20260810 06:25:25.251142 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.244683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38570 (local address 127.4.73.1:38423)
0810 06:25:25.245073 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.245082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.245103 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:25:25.245182 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.245189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.245410 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.245597 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:25.246006 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.247223 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.249133 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.249911 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.250021 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.250146 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.250901 (+ 755us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.250998 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":94,"threads_started":1}
I20260810 06:25:25.252228 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.244290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:41701 (local address 127.0.0.1:57870)
0810 06:25:25.245138 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:25.245149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:25.245222 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:25.246382 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:25.246385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:25.246403 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:25.246490 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.246496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.247469 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.247471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.249002 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.249009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.251132 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.251135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.251902 (+ 767us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.251927 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:25:25.252031 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 06:25:25.252104 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":178,"thread_start_us":138,"threads_started":1}
I20260810 06:25:25.252362 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.244752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.4.73.3:41701)
0810 06:25:25.245966 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.245984 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.246008 (+ 24us) server_negotiation.cc:407] Connection header received
0810 06:25:25.246092 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.246096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.246225 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.246406 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:25.246602 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.247348 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.250211 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.250784 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.250924 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.251067 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.252093 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.252194 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":162,"threads_started":1}
I20260810 06:25:25.253505 11797 tablet_service.cc:1467] Processing CreateTablet for tablet cbec9b566609493bbbc43475899751d3 (DEFAULT_TABLE table=test-table [id=ef5b740af2334376816cc21e4e69b746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:25.253786 11575 tablet_service.cc:1467] Processing CreateTablet for tablet cbec9b566609493bbbc43475899751d3 (DEFAULT_TABLE table=test-table [id=ef5b740af2334376816cc21e4e69b746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:25.254611 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbec9b566609493bbbc43475899751d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:25.254804 11575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbec9b566609493bbbc43475899751d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:25.254864 12013 tablet_service.cc:1467] Processing CreateTablet for tablet cbec9b566609493bbbc43475899751d3 (DEFAULT_TABLE table=test-table [id=ef5b740af2334376816cc21e4e69b746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:25.255770 12013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbec9b566609493bbbc43475899751d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:25.260123 12155 tablet_bootstrap.cc:492] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Bootstrap starting.
I20260810 06:25:25.262504 12155 tablet_bootstrap.cc:654] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:25.262820 12156 tablet_bootstrap.cc:492] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Bootstrap starting.
I20260810 06:25:25.263852 12155 log.cc:824] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:25.265048 12156 tablet_bootstrap.cc:654] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:25.266021 12156 log.cc:824] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:25.267066 12161 tablet_bootstrap.cc:492] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d: Bootstrap starting.
I20260810 06:25:25.268356 12156 tablet_bootstrap.cc:492] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: No bootstrap required, opened a new log
I20260810 06:25:25.268558 12156 ts_tablet_manager.cc:1397] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:25:25.269485 12161 tablet_bootstrap.cc:654] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:25.270982 12161 log.cc:824] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:25.271080 12155 tablet_bootstrap.cc:492] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: No bootstrap required, opened a new log
I20260810 06:25:25.271260 12155 ts_tablet_manager.cc:1397] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260810 06:25:25.272502 12156 raft_consensus.cc:348] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.272763 12156 raft_consensus.cc:374] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:25.272816 12156 raft_consensus.cc:729] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e3de33d9e14c348e3532f6a8d7a207, State: Initialized, Role: FOLLOWER
I20260810 06:25:25.273102 12161 tablet_bootstrap.cc:492] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d: No bootstrap required, opened a new log
I20260810 06:25:25.273263 12161 ts_tablet_manager.cc:1397] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:25:25.273284 12156 consensus_queue.cc:260] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.274278 12156 ts_tablet_manager.cc:1428] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:25:25.274611 12128 heartbeater.cc:503] Master 127.4.73.62:34719 was elected leader, sending a full tablet report...
I20260810 06:25:25.274894 12155 raft_consensus.cc:348] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.275147 12155 raft_consensus.cc:374] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:25.275195 12155 raft_consensus.cc:729] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2573b4f200645a49c84fc5f5e0dcfd1, State: Initialized, Role: FOLLOWER
I20260810 06:25:25.275585 12155 consensus_queue.cc:260] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.276546 12155 ts_tablet_manager.cc:1428] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 06:25:25.276906 11900 heartbeater.cc:503] Master 127.4.73.62:34719 was elected leader, sending a full tablet report...
I20260810 06:25:25.277419 12161 raft_consensus.cc:348] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.277743 12161 raft_consensus.cc:374] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:25.277797 12161 raft_consensus.cc:729] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09152cbf4f6a4ab1a01223fdbf831a5d, State: Initialized, Role: FOLLOWER
I20260810 06:25:25.278254 12161 consensus_queue.cc:260] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.280890 11682 heartbeater.cc:503] Master 127.4.73.62:34719 was elected leader, sending a full tablet report...
I20260810 06:25:25.281131 12161 ts_tablet_manager.cc:1428] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
W20260810 06:25:25.444211 12129 tablet.cc:2369] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:25.444391 12166 raft_consensus.cc:482] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:25.444595 12166 raft_consensus.cc:504] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.445785 12166 leader_election.cc:288] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09152cbf4f6a4ab1a01223fdbf831a5d (127.4.73.1:38423), b2e3de33d9e14c348e3532f6a8d7a207 (127.4.73.3:41701)
I20260810 06:25:25.449523 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.446166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:52093 (local address 127.4.73.3:41701)
0810 06:25:25.446286 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.446291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.446307 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:25:25.446438 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.446441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.446492 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.446596 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:25.446888 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.447389 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.448536 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.448943 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.449070 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.449135 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.449329 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.449404 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:25:25.449523 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.446044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:41701 (local address 127.4.73.2:52093)
0810 06:25:25.446177 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:25.446193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:25.446300 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:25.446613 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:25.446616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:25.446637 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:25.446764 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.446770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.447512 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.447519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.448398 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.448414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.449073 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.449077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.449186 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.449208 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:25.449308 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 06:25:25.449386 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 06:25:25.449944 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.446042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:38423 (local address 127.4.73.2:34553)
0810 06:25:25.446617 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:25.446628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:25.446742 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:25.447014 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:25.447016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:25.447033 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:25.447157 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.447162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.447906 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.447910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.448807 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:25.448818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.449482 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.449486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:25.449589 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.449619 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 06:25:25.449738 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 06:25:25.449800 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":275,"thread_start_us":119,"threads_started":1}
I20260810 06:25:25.449996 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.446138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:34553 (local address 127.4.73.1:38423)
0810 06:25:25.446266 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.446270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.446710 (+ 440us) server_negotiation.cc:407] Connection header received
0810 06:25:25.446856 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.446859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.446914 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.447001 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:25.447277 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.447776 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.448936 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.449356 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.449478 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.449538 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.449777 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.449863 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:25:25.450242 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec9b566609493bbbc43475899751d3" candidate_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" is_pre_election: true
I20260810 06:25:25.450598 12047 raft_consensus.cc:2454] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2573b4f200645a49c84fc5f5e0dcfd1 in term 0.
I20260810 06:25:25.450600 11603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec9b566609493bbbc43475899751d3" candidate_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" is_pre_election: true
I20260810 06:25:25.450875 11603 raft_consensus.cc:2454] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2573b4f200645a49c84fc5f5e0dcfd1 in term 0.
I20260810 06:25:25.451152 11718 leader_election.cc:302] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2573b4f200645a49c84fc5f5e0dcfd1, b2e3de33d9e14c348e3532f6a8d7a207; no voters:
I20260810 06:25:25.451445 12166 raft_consensus.cc:2781] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:25.451543 12166 raft_consensus.cc:482] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:25.451591 12166 raft_consensus.cc:3037] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:25.452692 12166 raft_consensus.cc:504] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.453109 12166 leader_election.cc:288] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [CANDIDATE]: Term 1 election: Requested vote from peers 09152cbf4f6a4ab1a01223fdbf831a5d (127.4.73.1:38423), b2e3de33d9e14c348e3532f6a8d7a207 (127.4.73.3:41701)
I20260810 06:25:25.453478 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec9b566609493bbbc43475899751d3" candidate_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e3de33d9e14c348e3532f6a8d7a207"
I20260810 06:25:25.453567 11603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbec9b566609493bbbc43475899751d3" candidate_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d"
I20260810 06:25:25.453584 12047 raft_consensus.cc:3037] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:25.453742 11603 raft_consensus.cc:3037] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:25.454825 12047 raft_consensus.cc:2454] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2573b4f200645a49c84fc5f5e0dcfd1 in term 1.
I20260810 06:25:25.455003 11603 raft_consensus.cc:2454] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2573b4f200645a49c84fc5f5e0dcfd1 in term 1.
I20260810 06:25:25.455183 11718 leader_election.cc:302] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2573b4f200645a49c84fc5f5e0dcfd1, b2e3de33d9e14c348e3532f6a8d7a207; no voters:
I20260810 06:25:25.455351 12166 raft_consensus.cc:2781] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:25.455574 12166 raft_consensus.cc:686] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 1 LEADER]: Becoming Leader. State: Replica: a2573b4f200645a49c84fc5f5e0dcfd1, State: Running, Role: LEADER
I20260810 06:25:25.455914 12166 consensus_queue.cc:237] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } }
I20260810 06:25:25.459161 11372 catalog_manager.cc:5495] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2573b4f200645a49c84fc5f5e0dcfd1 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2573b4f200645a49c84fc5f5e0dcfd1" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 } health_report { overall_health: UNKNOWN } } }
W20260810 06:25:25.467213 11901 tablet.cc:2369] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:25.495658 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.492381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.4.73.2:37315)
0810 06:25:25.492525 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.492529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.492543 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:25.492658 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.492661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.492716 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.492802 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:25.493065 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.493605 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.494379 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:25.494580 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:25.494674 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:25.494958 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:25.495340 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:25.495424 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.495485 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.495532 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 06:25:25.501235 11603 raft_consensus.cc:1261] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 1 FOLLOWER]: Refusing update from remote peer a2573b4f200645a49c84fc5f5e0dcfd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:25.501235 12047 raft_consensus.cc:1261] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 1 FOLLOWER]: Refusing update from remote peer a2573b4f200645a49c84fc5f5e0dcfd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:25.501929 12166 consensus_queue.cc:1035] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e3de33d9e14c348e3532f6a8d7a207" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:25.502130 12169 consensus_queue.cc:1035] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09152cbf4f6a4ab1a01223fdbf831a5d" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:25.507943 12174 mvcc.cc:205] Tried to move back new op lower bound from 7316861442046099456 to 7316861441870508032. Current Snapshot: MvccSnapshot[applied={T|T < 7316861442046099456}]
I20260810 06:25:25.510710 12177 mvcc.cc:205] Tried to move back new op lower bound from 7316861442046099456 to 7316861441870508032. Current Snapshot: MvccSnapshot[applied={T|T < 7316861442046099456 or (T in {7316861442046099456})}]
I20260810 06:25:25.516638 11372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:25.516824 11372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:25.519117 11372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a23a78f48e64af990a55b5bc077741a: Sending DeleteTablet for 3 replicas of tablet cbec9b566609493bbbc43475899751d3
I20260810 06:25:25.519862 11575 tablet_service.cc:1514] Processing DeleteTablet for tablet cbec9b566609493bbbc43475899751d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:25 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38570
I20260810 06:25:25.519924 12013 tablet_service.cc:1514] Processing DeleteTablet for tablet cbec9b566609493bbbc43475899751d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:25 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:57870
I20260810 06:25:25.519945 11797 tablet_service.cc:1514] Processing DeleteTablet for tablet cbec9b566609493bbbc43475899751d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:25 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:51428
I20260810 06:25:25.520526 12193 tablet_replica.cc:330] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d: stopping tablet replica
I20260810 06:25:25.520815 12193 raft_consensus.cc:2229] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:25.520865 12194 tablet_replica.cc:330] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: stopping tablet replica
I20260810 06:25:25.521103 12193 raft_consensus.cc:2258] T cbec9b566609493bbbc43475899751d3 P 09152cbf4f6a4ab1a01223fdbf831a5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:25.521124 12194 raft_consensus.cc:2229] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:25.521377 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 11468
I20260810 06:25:25.521435 12194 raft_consensus.cc:2258] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:25.529238 12190 tablet_replica.cc:330] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: stopping tablet replica
I20260810 06:25:25.529515 12190 raft_consensus.cc:2229] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:25.529793 12190 raft_consensus.cc:2258] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:25.531143 12194 ts_tablet_manager.cc:1905] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:25.533051 12190 ts_tablet_manager.cc:1905] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:25.534828 12194 ts_tablet_manager.cc:1918] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:25.534961 12194 log.cc:1196] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/cbec9b566609493bbbc43475899751d3
I20260810 06:25:25.535313 12194 ts_tablet_manager.cc:1939] T cbec9b566609493bbbc43475899751d3 P a2573b4f200645a49c84fc5f5e0dcfd1: Deleting consensus metadata
I20260810 06:25:25.535822 12190 ts_tablet_manager.cc:1918] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:25.535925 12190 log.cc:1196] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/cbec9b566609493bbbc43475899751d3
I20260810 06:25:25.536244 12190 ts_tablet_manager.cc:1939] T cbec9b566609493bbbc43475899751d3 P b2e3de33d9e14c348e3532f6a8d7a207: Deleting consensus metadata
I20260810 06:25:25.537401 11350 catalog_manager.cc:4841] TS b2e3de33d9e14c348e3532f6a8d7a207 (127.4.73.3:41701): tablet cbec9b566609493bbbc43475899751d3 (table test-table [id=ef5b740af2334376816cc21e4e69b746]) successfully deleted
I20260810 06:25:25.539846 11353 catalog_manager.cc:4841] TS a2573b4f200645a49c84fc5f5e0dcfd1 (127.4.73.2:37315): tablet cbec9b566609493bbbc43475899751d3 (table test-table [id=ef5b740af2334376816cc21e4e69b746]) successfully deleted
W20260810 06:25:25.541810 11354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:38423 (error 108)
I20260810 06:25:25.542899 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.542579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:38423 (local address 127.0.0.1:38574)
0810 06:25:25.542716 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:25.542787 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:38423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 06:25:25.543128 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 11691
W20260810 06:25:25.543107 11354 catalog_manager.cc:4584] TS 09152cbf4f6a4ab1a01223fdbf831a5d (127.4.73.1:38423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbec9b566609493bbbc43475899751d3: Network error: Client connection negotiation failed: client connection to 127.4.73.1:38423: connect: Connection refused (error 111)
I20260810 06:25:25.555128 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 11908
I20260810 06:25:25.564702 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 11320
2026-08-10T06:25:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4087 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T06:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:25Z Disabled control of system clock
I20260810 06:25:25.602851 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34927
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 06:25:25.711745 12202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:25.712008 12202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:25.712042 12202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:25.715919 12202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:25.715974 12202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:25.715991 12202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:25.716006 12202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:25.716022 12202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:25.720774 12202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:25.721709 12202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:25.723006 12202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:25:25.731477 12202 server_base.cc:1034] running on GCE node
W20260810 06:25:25.731413 12210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:25.731429 12211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:25.731423 12213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:25.732198 12202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:25.733304 12202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:25.734493 12202 hybrid_clock.cc:648] HybridClock initialized: now 1786343125734473 us; error 30 us; skew 500 ppm
I20260810 06:25:25.736346 12202 webserver.cc:460] Webserver started at http://127.4.73.62:43349/ using document root <none> and password file <none>
I20260810 06:25:25.736923 12202 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:25.737030 12202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:25.737253 12202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:25.739231 12202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "e26ed3a1815e42fdaae8a99872a02a18"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "b182ec8ffb824fd8874275ba23e06942"
server_key_iv: "17db8c5576a5b8858e2e2116520b5157"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.739806 12202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "e26ed3a1815e42fdaae8a99872a02a18"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "b182ec8ffb824fd8874275ba23e06942"
server_key_iv: "17db8c5576a5b8858e2e2116520b5157"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.743528 12202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:25:25.745867 12226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:25.747010 12202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:25.747153 12202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "e26ed3a1815e42fdaae8a99872a02a18"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "b182ec8ffb824fd8874275ba23e06942"
server_key_iv: "17db8c5576a5b8858e2e2116520b5157"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.747267 12202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:25.751748 12202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:25.752349 12202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:25.752533 12202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:25.758883 12202 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:34927
I20260810 06:25:25.758901 12321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:34927 every 8 connection(s)
I20260810 06:25:25.759701 12202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:25.763960 12323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:25.768177 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 12202
I20260810 06:25:25.768293 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:25.768610 4388 external_mini_cluster.cc:1428] Setting key 9ba8c6a5d1a865f2ad685f9009ca4368
I20260810 06:25:25.769724 12323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18: Bootstrap starting.
I20260810 06:25:25.771924 12323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:25.772806 12323 log.cc:824] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:25.774055 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:25.770006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.4.73.62:34927)
0810 06:25:25.770612 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:25.770626 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:25.770660 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:25:25.771244 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:25.771258 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:25.771585 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:25.771936 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:25.772764 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:25.772769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:25.772789 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:25.772799 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:25.772850 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:25.772865 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:25.773002 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:25.773127 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 06:25:25.773404 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260810 06:25:25.774926 12323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18: No bootstrap required, opened a new log
I20260810 06:25:25.778597 12323 raft_consensus.cc:348] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26ed3a1815e42fdaae8a99872a02a18" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34927 } }
I20260810 06:25:25.778796 12323 raft_consensus.cc:374] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:25.778829 12323 raft_consensus.cc:729] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26ed3a1815e42fdaae8a99872a02a18, State: Initialized, Role: FOLLOWER
I20260810 06:25:25.779217 12323 consensus_queue.cc:260] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26ed3a1815e42fdaae8a99872a02a18" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34927 } }
I20260810 06:25:25.779331 12323 raft_consensus.cc:388] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:25.779384 12323 raft_consensus.cc:482] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:25.779451 12323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:25.780216 12323 raft_consensus.cc:504] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26ed3a1815e42fdaae8a99872a02a18" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34927 } }
I20260810 06:25:25.780568 12323 leader_election.cc:302] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e26ed3a1815e42fdaae8a99872a02a18; no voters:
I20260810 06:25:25.780805 12323 leader_election.cc:288] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:25.780921 12331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:25.781167 12331 raft_consensus.cc:686] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [term 1 LEADER]: Becoming Leader. State: Replica: e26ed3a1815e42fdaae8a99872a02a18, State: Running, Role: LEADER
I20260810 06:25:25.781473 12331 consensus_queue.cc:237] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26ed3a1815e42fdaae8a99872a02a18" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34927 } }
I20260810 06:25:25.782030 12323 sys_catalog.cc:564] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:25.783063 12333 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e26ed3a1815e42fdaae8a99872a02a18. Latest consensus state: current_term: 1 leader_uuid: "e26ed3a1815e42fdaae8a99872a02a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26ed3a1815e42fdaae8a99872a02a18" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34927 } } }
I20260810 06:25:25.783212 12333 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:25.783479 12332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e26ed3a1815e42fdaae8a99872a02a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e26ed3a1815e42fdaae8a99872a02a18" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34927 } } }
I20260810 06:25:25.783582 12332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:25.783736 12343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:25.786864 12343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:25.792205 12343 catalog_manager.cc:1347] Generated new cluster ID: b5c42a958d3649a2b68a6f06bfe12c6f
I20260810 06:25:25.792272 12343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:25.812314 12343 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:25.812650 12343 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:25.822263 12343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18: Generated new TSK 0
I20260810 06:25:25.823026 12343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:25:25.836689 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34927
--builtin_ntp_servers=127.4.73.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 06:25:25.956409 12367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:25.956660 12367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:25.956696 12367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:25.960289 12367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:25.960337 12367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:25.960384 12367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:25.964820 12367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:25.965636 12367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:25.966833 12367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:25.975745 12379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:25.975756 12376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:25.975773 12375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:25.976145 12367 server_base.cc:1034] running on GCE node
I20260810 06:25:25.976610 12367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:25.977151 12367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:25.978310 12367 hybrid_clock.cc:648] HybridClock initialized: now 1786343125978276 us; error 48 us; skew 500 ppm
I20260810 06:25:25.980177 12367 webserver.cc:460] Webserver started at http://127.4.73.1:38597/ using document root <none> and password file <none>
I20260810 06:25:25.980762 12367 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:25.980824 12367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:25.980983 12367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:25.983008 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "6ebeacf637105c78f034de9e53a5eca2"
server_key_iv: "36353d55d72dbd679dc518299e755912"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.983705 12367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "6ebeacf637105c78f034de9e53a5eca2"
server_key_iv: "36353d55d72dbd679dc518299e755912"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.987164 12367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:25:25.989454 12390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:25.990590 12367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:25.990681 12367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7"
format_stamp: "Formatted at 2026-08-10 06:25:25 on dist-test-slave-45dx"
server_key: "6ebeacf637105c78f034de9e53a5eca2"
server_key_iv: "36353d55d72dbd679dc518299e755912"
server_key_version: "encryptionkey@0"
I20260810 06:25:25.990762 12367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:25.995075 12367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:25.995575 12367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:25.995707 12367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:25.996147 12367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:25.996999 12367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:25.997045 12367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:25.997081 12367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:25.997105 12367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:26.005542 12367 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:33019
I20260810 06:25:26.005570 12592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:33019 every 8 connection(s)
I20260810 06:25:26.006371 12367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:26.014000 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 12367
I20260810 06:25:26.014153 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:26.014405 4388 external_mini_cluster.cc:1428] Setting key 449486dc1d3a7652da1ef4b4798fc688
I20260810 06:25:26.016436 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.008130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:44071 (local address 127.4.73.62:34927)
0810 06:25:26.008275 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.008279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.009366 (+ 1087us) server_negotiation.cc:407] Connection header received
0810 06:25:26.010750 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.010753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.010798 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.010906 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.015676 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.015679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.015681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.015692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.015734 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.015746 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.015835 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.016215 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.016251 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 06:25:26.016640 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34927
--builtin_ntp_servers=127.4.73.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 06:25:26.017489 12595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.008562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34927 (local address 127.4.73.1:44071)
0810 06:25:26.009220 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.009256 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.010537 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.011067 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.011073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.011166 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.015388 (+ 4222us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.015390 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.015440 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.015521 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.016058 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.016067 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.016074 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.016249 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.016531 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
I20260810 06:25:26.018744 12593 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34927
I20260810 06:25:26.018957 12593 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:26.019359 12593 heartbeater.cc:511] Master 127.4.73.62:34927 requested a full tablet report, sending...
I20260810 06:25:26.020841 12257 ts_manager.cc:194] Registered new tserver with Master: d1f6e42e5f08493bbdb3e60d7a961ae7 (127.4.73.1:33019)
W20260810 06:25:26.127004 12598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:26.127238 12598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:26.127275 12598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:26.131266 12598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:26.131315 12598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:26.131402 12598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:26.136003 12598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:26.136968 12598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:26.138242 12598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:26.147893 12613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:26.147924 12611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:26.147962 12610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:26.147915 12598 server_base.cc:1034] running on GCE node
I20260810 06:25:26.148559 12598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:26.149181 12598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:26.150370 12598 hybrid_clock.cc:648] HybridClock initialized: now 1786343126150355 us; error 34 us; skew 500 ppm
I20260810 06:25:26.152096 12598 webserver.cc:460] Webserver started at http://127.4.73.2:35749/ using document root <none> and password file <none>
I20260810 06:25:26.152683 12598 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:26.152773 12598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:26.153046 12598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:26.154958 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "0314c831415249298c3010834b5e6a7a"
format_stamp: "Formatted at 2026-08-10 06:25:26 on dist-test-slave-45dx"
server_key: "58af3f8f7104690bcdbc43e6f6c509bf"
server_key_iv: "c6ae5b61ef8d31c617eb8fb78d9625d2"
server_key_version: "encryptionkey@0"
I20260810 06:25:26.155509 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "0314c831415249298c3010834b5e6a7a"
format_stamp: "Formatted at 2026-08-10 06:25:26 on dist-test-slave-45dx"
server_key: "58af3f8f7104690bcdbc43e6f6c509bf"
server_key_iv: "c6ae5b61ef8d31c617eb8fb78d9625d2"
server_key_version: "encryptionkey@0"
I20260810 06:25:26.159237 12598 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 06:25:26.161478 12622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:26.162600 12598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:26.162735 12598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "0314c831415249298c3010834b5e6a7a"
format_stamp: "Formatted at 2026-08-10 06:25:26 on dist-test-slave-45dx"
server_key: "58af3f8f7104690bcdbc43e6f6c509bf"
server_key_iv: "c6ae5b61ef8d31c617eb8fb78d9625d2"
server_key_version: "encryptionkey@0"
I20260810 06:25:26.162853 12598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:26.167153 12598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:26.167732 12598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:26.167922 12598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:26.168499 12598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:26.169335 12598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:26.169415 12598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:26.169486 12598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:26.169533 12598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:26.178090 12598 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:45001
I20260810 06:25:26.178112 12800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:45001 every 8 connection(s)
I20260810 06:25:26.179003 12598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:26.183497 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 12598
I20260810 06:25:26.183624 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:26.183985 4388 external_mini_cluster.cc:1428] Setting key 728515a55b2e4321e79669ccdcef2395
I20260810 06:25:26.185369 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.180758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:36941 (local address 127.4.73.62:34927)
0810 06:25:26.180890 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.180894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.181857 (+ 963us) server_negotiation.cc:407] Connection header received
0810 06:25:26.183336 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.183340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.183395 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.183499 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.184856 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.184860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.184863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.184875 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.184896 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.184905 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.185025 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.185188 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.185231 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:25:26.186034 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34927
--builtin_ntp_servers=127.4.73.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 06:25:26.186196 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.181058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34927 (local address 127.4.73.2:36941)
0810 06:25:26.181685 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.181724 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.183123 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.183704 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.183714 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.183872 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.184582 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.184585 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.184637 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.184716 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.185012 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.185021 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.185026 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.185217 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.185432 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
I20260810 06:25:26.187402 12802 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34927
I20260810 06:25:26.187626 12802 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:26.188016 12802 heartbeater.cc:511] Master 127.4.73.62:34927 requested a full tablet report, sending...
I20260810 06:25:26.189080 12257 ts_manager.cc:194] Registered new tserver with Master: 0314c831415249298c3010834b5e6a7a (127.4.73.2:45001)
W20260810 06:25:26.315341 12806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:26.315601 12806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:26.315639 12806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:26.319623 12806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:26.319674 12806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:26.319727 12806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:26.324445 12806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:34927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:26.325359 12806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:26.326581 12806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:26.336004 12819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:26.336025 12815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:26.336018 12814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:26.336284 12806 server_base.cc:1034] running on GCE node
I20260810 06:25:26.336815 12806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:26.337442 12806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:26.338632 12806 hybrid_clock.cc:648] HybridClock initialized: now 1786343126338621 us; error 37 us; skew 500 ppm
I20260810 06:25:26.340482 12806 webserver.cc:460] Webserver started at http://127.4.73.3:40415/ using document root <none> and password file <none>
I20260810 06:25:26.341071 12806 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:26.341159 12806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:26.341399 12806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:26.343485 12806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "dcac8e08db704b579706dc6792cfbb88"
format_stamp: "Formatted at 2026-08-10 06:25:26 on dist-test-slave-45dx"
server_key: "aac1baa8581de0c1a99e72445621eed1"
server_key_iv: "339955dc703b8e525d68362ffd2d40d5"
server_key_version: "encryptionkey@0"
I20260810 06:25:26.344121 12806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "dcac8e08db704b579706dc6792cfbb88"
format_stamp: "Formatted at 2026-08-10 06:25:26 on dist-test-slave-45dx"
server_key: "aac1baa8581de0c1a99e72445621eed1"
server_key_iv: "339955dc703b8e525d68362ffd2d40d5"
server_key_version: "encryptionkey@0"
I20260810 06:25:26.347692 12806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:25:26.350214 12829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:26.351435 12806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:25:26.351574 12806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "dcac8e08db704b579706dc6792cfbb88"
format_stamp: "Formatted at 2026-08-10 06:25:26 on dist-test-slave-45dx"
server_key: "aac1baa8581de0c1a99e72445621eed1"
server_key_iv: "339955dc703b8e525d68362ffd2d40d5"
server_key_version: "encryptionkey@0"
I20260810 06:25:26.351698 12806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:26.356228 12806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:26.356802 12806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:26.356988 12806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:26.357508 12806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:26.358364 12806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:26.358435 12806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:26.358502 12806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:26.358552 12806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:26.367666 12806 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:37077
I20260810 06:25:26.367697 13023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:37077 every 8 connection(s)
I20260810 06:25:26.368533 12806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:26.374038 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.370173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:40651 (local address 127.4.73.62:34927)
0810 06:25:26.370305 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.370308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.371182 (+ 874us) server_negotiation.cc:407] Connection header received
0810 06:25:26.372440 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.372442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.372519 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.372613 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.373674 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.373676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.373679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.373688 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.373703 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.373711 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.373781 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.373902 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.373938 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:25:26.374780 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.370455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34927 (local address 127.4.73.3:40651)
0810 06:25:26.371044 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.371082 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.372178 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.372754 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.372760 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.372838 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.373442 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.373444 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.373480 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.373542 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.373785 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.373792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.373796 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.373916 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.374094 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260810 06:25:26.375600 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 12806
I20260810 06:25:26.375733 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:26.375948 13024 heartbeater.cc:347] Connected to a master server at 127.4.73.62:34927
I20260810 06:25:26.375998 4388 external_mini_cluster.cc:1428] Setting key 80eb90827237caeb83b4586e7c0bc4fb
I20260810 06:25:26.376185 13024 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:26.376600 13024 heartbeater.cc:511] Master 127.4.73.62:34927 requested a full tablet report, sending...
I20260810 06:25:26.377671 12255 ts_manager.cc:194] Registered new tserver with Master: dcac8e08db704b579706dc6792cfbb88 (127.4.73.3:37077)
I20260810 06:25:26.389484 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:25:26.392817 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.391364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.4.73.62:34927)
0810 06:25:26.391485 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.391488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.391700 (+ 212us) server_negotiation.cc:407] Connection header received
0810 06:25:26.391788 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.391790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.391817 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.391873 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.392561 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.392563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.392565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.392574 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.392588 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.392595 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.392658 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.392700 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.392728 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:25:26.395741 12255 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:26.398823 12255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:26.410975 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.407517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.4.73.2:45001)
0810 06:25:26.408867 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.408880 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.408901 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:25:26.408948 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.408952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.409137 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.409663 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.410307 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.410314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.410328 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.410341 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.410389 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.410419 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.410569 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.410693 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.410771 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":119,"threads_started":1}
I20260810 06:25:26.411020 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.408197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.4.73.3:37077)
0810 06:25:26.408626 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.408640 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.408871 (+ 231us) server_negotiation.cc:407] Connection header received
0810 06:25:26.408975 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.408981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.409128 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.409275 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.410431 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.410439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.410455 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.410470 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.410511 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.410542 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.410618 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.410798 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.410868 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":197,"threads_started":1}
I20260810 06:25:26.411005 13045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.407341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:45001 (local address 127.0.0.1:60446)
0810 06:25:26.408070 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.408120 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.408222 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.409293 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.409296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.409367 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.410080 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.410083 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.410091 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.410166 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.410538 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.410548 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.410552 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.410663 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.410875 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":107,"threads_started":1}
I20260810 06:25:26.411525 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.408183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.4.73.1:33019)
0810 06:25:26.408509 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.408533 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.408882 (+ 349us) server_negotiation.cc:407] Connection header received
0810 06:25:26.408946 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.408952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.409137 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.409275 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.410676 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.410682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.410691 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.410702 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.410741 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.410766 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.410864 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.411272 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.411343 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":129,"threads_started":1}
I20260810 06:25:26.411592 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.407763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:37077 (local address 127.0.0.1:56560)
0810 06:25:26.408741 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.408757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.408882 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.409288 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.409294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.409367 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.410021 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.410023 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.410044 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.410303 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.410686 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.410689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.410691 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.411442 (+ 751us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.411483 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"thread_start_us":93,"threads_started":1}
I20260810 06:25:26.411170 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.407518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:33019 (local address 127.0.0.1:56686)
0810 06:25:26.408681 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.408699 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.408782 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.409973 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.409979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.409993 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.410374 (+ 381us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.410376 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.410387 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.410411 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.410908 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.410911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.410914 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.410987 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.411024 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":115,"threads_started":1}
I20260810 06:25:26.414098 12484 tablet_service.cc:1467] Processing CreateTablet for tablet b47a2f7f641f4e1babcbc0badc306dee (DEFAULT_TABLE table=test-table [id=fd95c1cba4c041798f2daac0b19e6449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:26.414270 12910 tablet_service.cc:1467] Processing CreateTablet for tablet b47a2f7f641f4e1babcbc0badc306dee (DEFAULT_TABLE table=test-table [id=fd95c1cba4c041798f2daac0b19e6449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:26.414280 12694 tablet_service.cc:1467] Processing CreateTablet for tablet b47a2f7f641f4e1babcbc0badc306dee (DEFAULT_TABLE table=test-table [id=fd95c1cba4c041798f2daac0b19e6449]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:26.415171 12484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47a2f7f641f4e1babcbc0badc306dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:26.415251 12910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47a2f7f641f4e1babcbc0badc306dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:26.415251 12694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47a2f7f641f4e1babcbc0badc306dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:26.421120 13052 tablet_bootstrap.cc:492] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: Bootstrap starting.
I20260810 06:25:26.422443 13054 tablet_bootstrap.cc:492] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Bootstrap starting.
I20260810 06:25:26.423569 13052 tablet_bootstrap.cc:654] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:26.424439 13052 log.cc:824] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:26.424530 13054 tablet_bootstrap.cc:654] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:26.424612 13053 tablet_bootstrap.cc:492] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Bootstrap starting.
I20260810 06:25:26.425269 13054 log.cc:824] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:26.426488 13052 tablet_bootstrap.cc:492] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: No bootstrap required, opened a new log
I20260810 06:25:26.426600 13053 tablet_bootstrap.cc:654] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:26.426681 13052 ts_tablet_manager.cc:1397] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:25:26.427209 13054 tablet_bootstrap.cc:492] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: No bootstrap required, opened a new log
I20260810 06:25:26.427284 13053 log.cc:824] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:26.427389 13054 ts_tablet_manager.cc:1397] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:25:26.429107 13053 tablet_bootstrap.cc:492] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: No bootstrap required, opened a new log
I20260810 06:25:26.429278 13053 ts_tablet_manager.cc:1397] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:25:26.431193 13052 raft_consensus.cc:348] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.431447 13052 raft_consensus.cc:374] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:26.431505 13052 raft_consensus.cc:729] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f6e42e5f08493bbdb3e60d7a961ae7, State: Initialized, Role: FOLLOWER
I20260810 06:25:26.431663 13054 raft_consensus.cc:348] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.431885 13054 raft_consensus.cc:374] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:26.431936 13054 raft_consensus.cc:729] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0314c831415249298c3010834b5e6a7a, State: Initialized, Role: FOLLOWER
I20260810 06:25:26.431912 13052 consensus_queue.cc:260] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.432351 13054 consensus_queue.cc:260] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.432693 12593 heartbeater.cc:503] Master 127.4.73.62:34927 was elected leader, sending a full tablet report...
I20260810 06:25:26.432905 13052 ts_tablet_manager.cc:1428] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 06:25:26.432866 13053 raft_consensus.cc:348] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.433099 13053 raft_consensus.cc:374] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:26.433156 13053 raft_consensus.cc:729] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcac8e08db704b579706dc6792cfbb88, State: Initialized, Role: FOLLOWER
I20260810 06:25:26.433218 13054 ts_tablet_manager.cc:1428] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 06:25:26.433300 12802 heartbeater.cc:503] Master 127.4.73.62:34927 was elected leader, sending a full tablet report...
I20260810 06:25:26.433553 13053 consensus_queue.cc:260] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.434435 13053 ts_tablet_manager.cc:1428] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:25:26.434461 13024 heartbeater.cc:503] Master 127.4.73.62:34927 was elected leader, sending a full tablet report...
W20260810 06:25:26.507524 12594 tablet.cc:2369] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:25:26.619526 13025 tablet.cc:2369] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:25:26.680383 12803 tablet.cc:2369] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:26.714852 13064 raft_consensus.cc:482] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:26.715129 13064 raft_consensus.cc:504] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.716447 13064 leader_election.cc:288] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0314c831415249298c3010834b5e6a7a (127.4.73.2:45001), d1f6e42e5f08493bbdb3e60d7a961ae7 (127.4.73.1:33019)
I20260810 06:25:26.718487 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.716897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:58659 (local address 127.4.73.1:33019)
0810 06:25:26.717030 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.717034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.717221 (+ 187us) server_negotiation.cc:407] Connection header received
0810 06:25:26.717366 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.717369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.717414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.717526 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.718079 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.718082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.718085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.718095 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.718111 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.718120 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.718214 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.718301 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.718343 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:25:26.718493 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.716780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:56375 (local address 127.4.73.2:45001)
0810 06:25:26.716907 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.716912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.716934 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:25:26.717107 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.717110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.717148 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.717244 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.718053 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.718059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.718061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.718076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.718103 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.718115 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.718214 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.718311 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.718349 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:25:26.718497 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.716683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:45001 (local address 127.4.73.3:56375)
0810 06:25:26.716869 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.716888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.716982 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.717270 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.717273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.717289 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.717799 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.717801 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.717828 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.717876 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.718212 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.718215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.718219 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.718299 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.718350 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 06:25:26.718519 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.716753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:33019 (local address 127.4.73.3:58659)
0810 06:25:26.717116 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.717128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:26.717249 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:26.717543 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:26.717545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:26.717556 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:26.717899 (+ 343us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:25:26.717903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:26.717914 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:25:26.717940 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:26.718212 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:26.718215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:26.718219 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:25:26.718299 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:25:26.718370 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":26,"thread_start_us":139,"threads_started":1}
I20260810 06:25:26.719238 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47a2f7f641f4e1babcbc0badc306dee" candidate_uuid: "dcac8e08db704b579706dc6792cfbb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0314c831415249298c3010834b5e6a7a" is_pre_election: true
I20260810 06:25:26.719399 12518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47a2f7f641f4e1babcbc0badc306dee" candidate_uuid: "dcac8e08db704b579706dc6792cfbb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" is_pre_election: true
I20260810 06:25:26.719573 12722 raft_consensus.cc:2454] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcac8e08db704b579706dc6792cfbb88 in term 0.
I20260810 06:25:26.719645 12518 raft_consensus.cc:2454] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcac8e08db704b579706dc6792cfbb88 in term 0.
I20260810 06:25:26.720114 12834 leader_election.cc:302] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f6e42e5f08493bbdb3e60d7a961ae7, dcac8e08db704b579706dc6792cfbb88; no voters:
I20260810 06:25:26.720382 13064 raft_consensus.cc:2781] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:26.720502 13064 raft_consensus.cc:482] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:26.720562 13064 raft_consensus.cc:3037] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:26.721637 13064 raft_consensus.cc:504] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.721992 13064 leader_election.cc:288] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [CANDIDATE]: Term 1 election: Requested vote from peers 0314c831415249298c3010834b5e6a7a (127.4.73.2:45001), d1f6e42e5f08493bbdb3e60d7a961ae7 (127.4.73.1:33019)
I20260810 06:25:26.722333 12518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47a2f7f641f4e1babcbc0badc306dee" candidate_uuid: "dcac8e08db704b579706dc6792cfbb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7"
I20260810 06:25:26.722359 12722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47a2f7f641f4e1babcbc0badc306dee" candidate_uuid: "dcac8e08db704b579706dc6792cfbb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0314c831415249298c3010834b5e6a7a"
I20260810 06:25:26.722465 12722 raft_consensus.cc:3037] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:26.722465 12518 raft_consensus.cc:3037] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:26.723826 12518 raft_consensus.cc:2454] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcac8e08db704b579706dc6792cfbb88 in term 1.
I20260810 06:25:26.723850 12722 raft_consensus.cc:2454] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcac8e08db704b579706dc6792cfbb88 in term 1.
I20260810 06:25:26.724133 12834 leader_election.cc:302] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f6e42e5f08493bbdb3e60d7a961ae7, dcac8e08db704b579706dc6792cfbb88; no voters:
I20260810 06:25:26.724362 13064 raft_consensus.cc:2781] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:26.724584 13064 raft_consensus.cc:686] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 1 LEADER]: Becoming Leader. State: Replica: dcac8e08db704b579706dc6792cfbb88, State: Running, Role: LEADER
I20260810 06:25:26.724912 13064 consensus_queue.cc:237] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } }
I20260810 06:25:26.728199 12248 catalog_manager.cc:5495] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcac8e08db704b579706dc6792cfbb88 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "dcac8e08db704b579706dc6792cfbb88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcac8e08db704b579706dc6792cfbb88" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37077 } health_report { overall_health: HEALTHY } } }
I20260810 06:25:26.771843 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.770276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.4.73.3:37077)
0810 06:25:26.770423 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:26.770428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:26.770446 (+ 18us) server_negotiation.cc:407] Connection header received
0810 06:25:26.770554 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:26.770558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:26.770624 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:26.770712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:26.771436 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:26.771439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:26.771442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:26.771454 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:26.771475 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:26.771485 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:26.771573 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:26.771671 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 06:25:26.771711 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:26.777539 12518 raft_consensus.cc:1261] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7 [term 1 FOLLOWER]: Refusing update from remote peer dcac8e08db704b579706dc6792cfbb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:26.777539 12722 raft_consensus.cc:1261] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 1 FOLLOWER]: Refusing update from remote peer dcac8e08db704b579706dc6792cfbb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:26.778281 13064 consensus_queue.cc:1035] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1f6e42e5f08493bbdb3e60d7a961ae7" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:26.778537 13066 consensus_queue.cc:1035] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0314c831415249298c3010834b5e6a7a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:26.785817 13073 mvcc.cc:205] Tried to move back new op lower bound from 7316861447273553920 to 7316861447068033024. Current Snapshot: MvccSnapshot[applied={T|T < 7316861447273553920}]
I20260810 06:25:26.788549 13072 mvcc.cc:205] Tried to move back new op lower bound from 7316861447273553920 to 7316861447068033024. Current Snapshot: MvccSnapshot[applied={T|T < 7316861447273553920}]
I20260810 06:25:26.794193 12257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:26.794384 12257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:26.796949 12257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e26ed3a1815e42fdaae8a99872a02a18: Sending DeleteTablet for 3 replicas of tablet b47a2f7f641f4e1babcbc0badc306dee
I20260810 06:25:26.797694 12694 tablet_service.cc:1514] Processing DeleteTablet for tablet b47a2f7f641f4e1babcbc0badc306dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:26 UTC) from {username='slave'} at 127.0.0.1:60446
I20260810 06:25:26.797773 12484 tablet_service.cc:1514] Processing DeleteTablet for tablet b47a2f7f641f4e1babcbc0badc306dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:26 UTC) from {username='slave'} at 127.0.0.1:56686
I20260810 06:25:26.798233 13088 tablet_replica.cc:330] T b47a2f7f641f4e1babcbc0badc306dee P d1f6e42e5f08493bbdb3e60d7a961ae7: stopping tablet replica
I20260810 06:25:26.798507 12910 tablet_service.cc:1514] Processing DeleteTablet for tablet b47a2f7f641f4e1babcbc0badc306dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:26 UTC) from {username='slave'} at 127.0.0.1:56560
I20260810 06:25:26.798593 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 12367
I20260810 06:25:26.813227 13090 tablet_replica.cc:330] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: stopping tablet replica
I20260810 06:25:26.813495 13087 tablet_replica.cc:330] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: stopping tablet replica
I20260810 06:25:26.813673 13090 raft_consensus.cc:2229] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:26.813778 13087 raft_consensus.cc:2229] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:26.814075 13090 raft_consensus.cc:2258] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:26.814154 13087 raft_consensus.cc:2258] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:26.815948 13090 ts_tablet_manager.cc:1905] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:26.816074 13087 ts_tablet_manager.cc:1905] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:26.818910 13087 ts_tablet_manager.cc:1918] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:26.818910 13090 ts_tablet_manager.cc:1918] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:26.819016 13090 log.cc:1196] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/b47a2f7f641f4e1babcbc0badc306dee
I20260810 06:25:26.819026 13087 log.cc:1196] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/b47a2f7f641f4e1babcbc0badc306dee
I20260810 06:25:26.819388 13090 ts_tablet_manager.cc:1939] T b47a2f7f641f4e1babcbc0badc306dee P dcac8e08db704b579706dc6792cfbb88: Deleting consensus metadata
I20260810 06:25:26.819388 13087 ts_tablet_manager.cc:1939] T b47a2f7f641f4e1babcbc0badc306dee P 0314c831415249298c3010834b5e6a7a: Deleting consensus metadata
I20260810 06:25:26.820694 12228 catalog_manager.cc:4841] TS 0314c831415249298c3010834b5e6a7a (127.4.73.2:45001): tablet b47a2f7f641f4e1babcbc0badc306dee (table test-table [id=fd95c1cba4c041798f2daac0b19e6449]) successfully deleted
I20260810 06:25:26.820703 12233 catalog_manager.cc:4841] TS dcac8e08db704b579706dc6792cfbb88 (127.4.73.3:37077): tablet b47a2f7f641f4e1babcbc0badc306dee (table test-table [id=fd95c1cba4c041798f2daac0b19e6449]) successfully deleted
W20260810 06:25:26.820755 12231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:33019 (error 108)
I20260810 06:25:26.821682 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 12598
I20260810 06:25:26.826510 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:26.821630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:33019 (local address 127.0.0.1:56688)
0810 06:25:26.826311 (+ 4681us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:26.826408 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:33019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 06:25:26.826737 12231 catalog_manager.cc:4584] TS d1f6e42e5f08493bbdb3e60d7a961ae7 (127.4.73.1:33019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b47a2f7f641f4e1babcbc0badc306dee: Network error: Client connection negotiation failed: client connection to 127.4.73.1:33019: connect: Connection refused (error 111)
I20260810 06:25:26.833676 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 12806
I20260810 06:25:26.846338 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 12202
2026-08-10T06:25:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1278 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 06:25:26.872864 4388 mini_oidc.cc:152] Starting JWKS server
I20260810 06:25:26.874626 4388 webserver.cc:460] Webserver started at https://127.0.0.1:46473/ using document root <none> and password file <none>
I20260810 06:25:26.874812 4388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 06:25:26.875088 4388 webserver.cc:460] Webserver started at http://127.0.0.1:42495/ using document root <none> and password file <none>
2026-08-10T06:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:26Z Disabled control of system clock
I20260810 06:25:26.885051 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:40049
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:40049
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46473/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/testchainca.cert with env {}
W20260810 06:25:26.992038 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:26.992333 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:26.992475 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:26.996340 13099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:26.996428 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:26.996457 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:26.996495 13099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:26.996531 13099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:25:26.996567 13099 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:25:26.996598 13099 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46473/jwks.json
I20260810 06:25:27.001528 13099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:40049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46473/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:27.002496 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:27.003479 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:25:27.011806 13099 server_base.cc:1034] running on GCE node
W20260810 06:25:27.011744 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:27.011767 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:27.011756 13107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:27.012717 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:27.013768 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:27.014931 13099 hybrid_clock.cc:648] HybridClock initialized: now 1786343127014903 us; error 40 us; skew 500 ppm
I20260810 06:25:27.016927 13099 webserver.cc:460] Webserver started at http://127.4.73.62:35899/ using document root <none> and password file <none>
I20260810 06:25:27.017503 13099 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:27.017570 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:27.017804 13099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:27.019869 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "7f82b1d0294a4ba8a7382667b20e672b"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "f529dc863e1ec86182fa301ddcdc5c43"
server_key_iv: "728e644e4f1ba6e0720335d51c910381"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.020471 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "7f82b1d0294a4ba8a7382667b20e672b"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "f529dc863e1ec86182fa301ddcdc5c43"
server_key_iv: "728e644e4f1ba6e0720335d51c910381"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.023979 13099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:27.026489 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:27.027694 13099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:27.027834 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "7f82b1d0294a4ba8a7382667b20e672b"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "f529dc863e1ec86182fa301ddcdc5c43"
server_key_iv: "728e644e4f1ba6e0720335d51c910381"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.027947 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:27.040623 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:27.041234 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:27.041421 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:27.048151 13099 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:40049
I20260810 06:25:27.048161 13207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:40049 every 8 connection(s)
I20260810 06:25:27.049103 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:27.050352 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 13099
I20260810 06:25:27.050518 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:27.050822 4388 external_mini_cluster.cc:1428] Setting key df03f6ac1434e24ba8d01a37f6f67669
I20260810 06:25:27.052907 13208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:27.059499 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b: Bootstrap starting.
I20260810 06:25:27.060815 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:27.052048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.4.73.62:40049)
0810 06:25:27.052727 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:27.052746 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:27.052788 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:25:27.053607 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:27.053629 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:27.054095 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:27.054443 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:27.054835 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.056269 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.057170 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.057874 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.058027 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.058857 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:27.058864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:27.058891 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:27.059342 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:27.059410 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:27.059436 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:27.059534 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:27.059665 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 06:25:27.059923 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":114,"threads_started":1}
I20260810 06:25:27.061918 13208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:27.062786 13208 log.cc:824] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:27.063659 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:25:27.064879 13208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b: No bootstrap required, opened a new log
I20260810 06:25:27.068245 13208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f82b1d0294a4ba8a7382667b20e672b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40049 } }
I20260810 06:25:27.068578 13208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:27.068655 13208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f82b1d0294a4ba8a7382667b20e672b, State: Initialized, Role: FOLLOWER
I20260810 06:25:27.069108 13208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f82b1d0294a4ba8a7382667b20e672b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40049 } }
I20260810 06:25:27.069291 13208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:27.069372 13208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:27.069468 13208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:27.070277 13208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f82b1d0294a4ba8a7382667b20e672b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40049 } }
I20260810 06:25:27.070605 13208 leader_election.cc:302] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f82b1d0294a4ba8a7382667b20e672b; no voters:
I20260810 06:25:27.070909 13208 leader_election.cc:288] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:27.070997 13221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:27.071240 13221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [term 1 LEADER]: Becoming Leader. State: Replica: 7f82b1d0294a4ba8a7382667b20e672b, State: Running, Role: LEADER
I20260810 06:25:27.071539 13221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f82b1d0294a4ba8a7382667b20e672b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40049 } }
I20260810 06:25:27.072064 13208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:27.073190 13223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f82b1d0294a4ba8a7382667b20e672b. Latest consensus state: current_term: 1 leader_uuid: "7f82b1d0294a4ba8a7382667b20e672b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f82b1d0294a4ba8a7382667b20e672b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40049 } } }
I20260810 06:25:27.073364 13223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:27.073678 13222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f82b1d0294a4ba8a7382667b20e672b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f82b1d0294a4ba8a7382667b20e672b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40049 } } }
I20260810 06:25:27.073772 13222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:27.073856 13234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:27.076860 13234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:27.082345 13234 catalog_manager.cc:1347] Generated new cluster ID: b5f964776fca425590d27e9793227386
I20260810 06:25:27.082417 13234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:27.098132 13234 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:27.099025 13234 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:27.115500 13234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f82b1d0294a4ba8a7382667b20e672b: Generated new TSK 0
I20260810 06:25:27.116179 13234 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 06:25:27.158613 13252 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 06:25:27.158761 13252 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 06:25:27.159373 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:27.134821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.4.73.62:40049)
0810 06:25:27.134942 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:27.134947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:27.135159 (+ 212us) server_negotiation.cc:407] Connection header received
0810 06:25:27.135391 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:27.135394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:27.135446 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:27.135544 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:25:27.135850 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.136429 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.137372 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.137598 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.137711 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.138131 (+ 420us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:25:27.158980 (+ 20849us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 06:25:27.159240 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58204: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 06:25:27.159432 13217 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58204: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 06:25:27.160844 13251 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:27.134746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:40049 (local address 127.0.0.1:58204)
0810 06:25:27.135073 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:27.135086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:27.135227 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:27.135577 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:27.135580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:27.135589 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:25:27.135716 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:27.135740 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.136584 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.136588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:27.137243 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:27.137251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.137704 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.137722 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:27.137832 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.137882 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:25:27.159220 (+ 21338us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 06:25:27.159474 (+ 254us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.73.62:40049: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.73.62:40049: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46473/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 06:25:27.161957 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 13099
2026-08-10T06:25:27Z chronyd exiting
I20260810 06:25:27.208202 4388 test_util.cc:175] -----------------------------------------------
I20260810 06:25:27.208289 4388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786343096561320-4388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (337 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 06:25:27.209470 4388 mini_oidc.cc:152] Starting JWKS server
I20260810 06:25:27.210343 4388 webserver.cc:460] Webserver started at https://127.0.0.1:45509/ using document root <none> and password file <none>
I20260810 06:25:27.210464 4388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 06:25:27.210781 4388 webserver.cc:460] Webserver started at http://127.0.0.1:44243/ using document root <none> and password file <none>
2026-08-10T06:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:27Z Disabled control of system clock
I20260810 06:25:27.221863 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:36051
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:42309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:36051
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45509/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/testchainca.cert with env {}
W20260810 06:25:27.335228 13261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:27.335493 13261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:27.335533 13261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:27.339438 13261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:27.339489 13261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:27.339504 13261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:27.339520 13261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:27.339535 13261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:25:27.339550 13261 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:25:27.339563 13261 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45509/jwks.json
I20260810 06:25:27.344812 13261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:36051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:36051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45509/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:27.345863 13261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:27.346819 13261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:27.355136 13272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:27.355099 13270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:27.355077 13268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:27.355325 13261 server_base.cc:1034] running on GCE node
I20260810 06:25:27.356113 13261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:27.357318 13261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:27.358513 13261 hybrid_clock.cc:648] HybridClock initialized: now 1786343127358488 us; error 36 us; skew 500 ppm
I20260810 06:25:27.360280 13261 webserver.cc:460] Webserver started at http://127.4.73.62:33435/ using document root <none> and password file <none>
I20260810 06:25:27.360903 13261 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:27.360993 13261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:27.361195 13261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:27.363210 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "19bec44a1a8242809e74aa10e9cd2fe3"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "0266e3ef85081db26b792cbf8a66d9e3"
server_key_iv: "f95253d020a04192b0e9b22c8783b4b8"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.363802 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "19bec44a1a8242809e74aa10e9cd2fe3"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "0266e3ef85081db26b792cbf8a66d9e3"
server_key_iv: "f95253d020a04192b0e9b22c8783b4b8"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.367406 13261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 06:25:27.369918 13283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:27.371157 13261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:27.371301 13261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "19bec44a1a8242809e74aa10e9cd2fe3"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "0266e3ef85081db26b792cbf8a66d9e3"
server_key_iv: "f95253d020a04192b0e9b22c8783b4b8"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.371411 13261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:27.398574 13261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:27.399317 13261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:27.399523 13261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:27.407040 13261 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:36051
I20260810 06:25:27.407071 13366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:36051 every 8 connection(s)
I20260810 06:25:27.407981 13261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:27.411257 13367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:27.416684 13367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3: Bootstrap starting.
I20260810 06:25:27.417943 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 13261
I20260810 06:25:27.418076 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:27.418363 4388 external_mini_cluster.cc:1428] Setting key 284cc9c5af22379841530695a04cf3c9
I20260810 06:25:27.419552 13367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:27.420339 13367 log.cc:824] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:27.422433 13367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3: No bootstrap required, opened a new log
I20260810 06:25:27.426642 13367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 36051 } }
I20260810 06:25:27.426892 13367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:27.426977 13367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bec44a1a8242809e74aa10e9cd2fe3, State: Initialized, Role: FOLLOWER
I20260810 06:25:27.427417 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:27.419430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.4.73.62:36051)
0810 06:25:27.420022 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:27.420035 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:27.420069 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:25:27.420784 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:27.420808 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:27.421233 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:27.421498 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:27.421829 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.422827 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.424016 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.424543 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.424658 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.425531 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:27.425536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:27.425547 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:27.425930 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:27.425985 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:27.426005 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:27.426081 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:27.426245 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 06:25:27.426530 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":126,"threads_started":1}
I20260810 06:25:27.427449 13367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 36051 } }
I20260810 06:25:27.427642 13367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:27.427732 13367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:27.427834 13367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:27.429219 13367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 36051 } }
I20260810 06:25:27.429587 13367 leader_election.cc:302] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19bec44a1a8242809e74aa10e9cd2fe3; no voters:
I20260810 06:25:27.429955 13367 leader_election.cc:288] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:27.430022 13374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:27.430231 13374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [term 1 LEADER]: Becoming Leader. State: Replica: 19bec44a1a8242809e74aa10e9cd2fe3, State: Running, Role: LEADER
I20260810 06:25:27.430539 13374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 36051 } }
I20260810 06:25:27.430600 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:25:27.431212 13367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:27.432168 13375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19bec44a1a8242809e74aa10e9cd2fe3. Latest consensus state: current_term: 1 leader_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 36051 } } }
I20260810 06:25:27.432248 13376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19bec44a1a8242809e74aa10e9cd2fe3" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 36051 } } }
I20260810 06:25:27.432337 13375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:27.432344 13376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:27.432749 13385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:27.435897 13385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:27.441329 13385 catalog_manager.cc:1347] Generated new cluster ID: 48ca6ce21fb440f2a81c079b19feecb9
I20260810 06:25:27.441401 13385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:27.465790 13385 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:27.466706 13385 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:27.474336 13385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19bec44a1a8242809e74aa10e9cd2fe3: Generated new TSK 0
I20260810 06:25:27.475049 13385 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 06:25:27.506630 13411 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 06:25:27.506726 13411 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 06:25:27.507359 13371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:27.489392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.4.73.62:36051)
0810 06:25:27.489524 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:27.489529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:27.489678 (+ 149us) server_negotiation.cc:407] Connection header received
0810 06:25:27.489854 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:27.489857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:27.489913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:27.490025 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:25:27.490231 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.491062 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.491933 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.492273 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.492406 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.492678 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:25:27.506972 (+ 14294us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 06:25:27.507213 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 06:25:27.507423 13371 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53572: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 06:25:27.507436 13408 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:27.489290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:36051 (local address 127.0.0.1:53572)
0810 06:25:27.489591 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:27.489603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:27.489702 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:27.490029 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:27.490032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:27.490038 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:25:27.490129 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:27.490134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.491163 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.491166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:27.491798 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:27.491805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.492396 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.492413 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:27.492487 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.492508 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:25:27.507156 (+ 14648us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 06:25:27.507279 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.73.62:36051: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
I20260810 06:25:27.507840 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 13261
2026-08-10T06:25:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (348 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 06:25:27.557513 4388 mini_oidc.cc:152] Starting JWKS server
I20260810 06:25:27.558377 4388 webserver.cc:460] Webserver started at https://127.0.0.1:33319/ using document root <none> and password file <none>
I20260810 06:25:27.558493 4388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 06:25:27.558777 4388 webserver.cc:460] Webserver started at http://127.0.0.1:40251/ using document root <none> and password file <none>
2026-08-10T06:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:27Z Disabled control of system clock
I20260810 06:25:27.568548 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46875
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:46875
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33319/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 06:25:27.678176 13421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:27.678438 13421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:27.678478 13421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:27.682469 13421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:27.682525 13421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:27.682541 13421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:27.682557 13421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:27.682574 13421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:25:27.682590 13421 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:25:27.682602 13421 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33319/jwks.json
I20260810 06:25:27.687197 13421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:46875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33319/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:27.688165 13421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:27.689558 13421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:27.698119 13434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:27.698153 13421 server_base.cc:1034] running on GCE node
W20260810 06:25:27.698347 13432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:27.698086 13436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:27.698922 13421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:27.699874 13421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:27.701033 13421 hybrid_clock.cc:648] HybridClock initialized: now 1786343127701014 us; error 30 us; skew 500 ppm
I20260810 06:25:27.702764 13421 webserver.cc:460] Webserver started at http://127.4.73.62:41289/ using document root <none> and password file <none>
I20260810 06:25:27.703317 13421 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:27.703380 13421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:27.703675 13421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:27.705693 13421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "5fbd133fd49b4007a5e19da8005c2a0e"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "46eebd795d297448bd04c7668ac4d7ff"
server_key_iv: "c1ac58de14065d4f1a56ff56db6c28d3"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.706264 13421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "5fbd133fd49b4007a5e19da8005c2a0e"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "46eebd795d297448bd04c7668ac4d7ff"
server_key_iv: "c1ac58de14065d4f1a56ff56db6c28d3"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.709789 13421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:25:27.712030 13448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:27.713189 13421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:27.713322 13421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "5fbd133fd49b4007a5e19da8005c2a0e"
format_stamp: "Formatted at 2026-08-10 06:25:27 on dist-test-slave-45dx"
server_key: "46eebd795d297448bd04c7668ac4d7ff"
server_key_iv: "c1ac58de14065d4f1a56ff56db6c28d3"
server_key_version: "encryptionkey@0"
I20260810 06:25:27.713451 13421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:27.747812 13421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:27.748679 13421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:27.748896 13421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:27.755811 13421 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:46875
I20260810 06:25:27.755824 13529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:46875 every 8 connection(s)
I20260810 06:25:27.756843 13421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:27.760154 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:27.764599 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 13421
I20260810 06:25:27.764705 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:27.764986 4388 external_mini_cluster.cc:1428] Setting key 6cc4975377035e62972eed4ca0eefdd5
I20260810 06:25:27.766213 13530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e: Bootstrap starting.
I20260810 06:25:27.768800 13530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:27.769625 13530 log.cc:824] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:27.771936 13530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e: No bootstrap required, opened a new log
I20260810 06:25:27.774573 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:27.766292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.4.73.62:46875)
0810 06:25:27.766955 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:27.766972 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:27.767008 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:25:27.767647 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:27.767664 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:27.768074 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:27.768433 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:27.768795 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.769945 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.771077 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.771531 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.771648 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.772572 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:27.772577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:27.772589 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:27.773172 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:27.773243 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:27.773272 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:27.773365 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:27.773631 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 06:25:27.773893 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":126,"threads_started":1}
I20260810 06:25:27.776217 13530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46875 } }
I20260810 06:25:27.776523 13530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:27.776603 13530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fbd133fd49b4007a5e19da8005c2a0e, State: Initialized, Role: FOLLOWER
I20260810 06:25:27.777052 13530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46875 } }
I20260810 06:25:27.777208 13530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:27.777294 13530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:27.777437 13530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:27.777835 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:25:27.778407 13530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46875 } }
I20260810 06:25:27.778756 13530 leader_election.cc:302] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fbd133fd49b4007a5e19da8005c2a0e; no voters:
I20260810 06:25:27.779062 13530 leader_election.cc:288] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:27.779196 13539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:27.779456 13539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [term 1 LEADER]: Becoming Leader. State: Replica: 5fbd133fd49b4007a5e19da8005c2a0e, State: Running, Role: LEADER
I20260810 06:25:27.779762 13539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46875 } }
I20260810 06:25:27.780234 13530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:27.781344 13540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46875 } } }
I20260810 06:25:27.781484 13540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:27.781800 13542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fbd133fd49b4007a5e19da8005c2a0e. Latest consensus state: current_term: 1 leader_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fbd133fd49b4007a5e19da8005c2a0e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46875 } } }
I20260810 06:25:27.781908 13542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:27.782034 13554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:27.784847 13554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:27.790221 13554 catalog_manager.cc:1347] Generated new cluster ID: dda977d41afd4dcca102a4b44251e5f5
I20260810 06:25:27.790311 13554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:27.805570 13554 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:27.806713 13554 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:27.815430 13554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fbd133fd49b4007a5e19da8005c2a0e: Generated new TSK 0
I20260810 06:25:27.816202 13554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:25:27.838032 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:27.832407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.4.73.62:46875)
0810 06:25:27.832540 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:27.832545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:27.832698 (+ 153us) server_negotiation.cc:407] Connection header received
0810 06:25:27.832826 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:27.832830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:27.832893 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:27.833009 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:25:27.833233 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.834053 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.835357 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.835711 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.835818 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.836080 (+ 262us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:25:27.837506 (+ 1426us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 06:25:27.837898 (+ 392us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 06:25:27.838104 13537 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 06:25:27.839540 13581 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:27.832267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:46875 (local address 127.0.0.1:55928)
0810 06:25:27.832613 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:27.832626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:27.832723 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:27.833018 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:27.833020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:27.833027 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:25:27.833130 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:27.833135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.834226 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.834233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:27.835240 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:27.835251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:27.835815 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:27.835831 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:27.835895 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:27.835917 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:25:27.837690 (+ 1773us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 06:25:27.839353 (+ 1663us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.73.62:46875: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33319/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":1464,"thread_start_us":114,"threads_started":1}
I20260810 06:25:27.840221 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 13421
E20260810 06:25:27.852787 13584 webserver.cc:533] Webserver: sslize failed:
E20260810 06:25:27.852876 13584 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T06:25:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T06:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:27Z Disabled control of system clock
I20260810 06:25:27.894886 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34417
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34417
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/testchainca.cert with env {}
W20260810 06:25:28.003211 13593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:28.003433 13593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:28.003469 13593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:28.007256 13593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:28.007309 13593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:28.007324 13593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:28.007340 13593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:28.007355 13593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 06:25:28.007411 13593 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 06:25:28.007431 13593 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 06:25:28.013047 13593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:28.013960 13593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:28.014828 13593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:28.023118 13603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:28.023159 13593 server_base.cc:1034] running on GCE node
W20260810 06:25:28.023126 13610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:28.023126 13602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:28.023973 13593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:28.025023 13593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:28.026188 13593 hybrid_clock.cc:648] HybridClock initialized: now 1786343128026163 us; error 36 us; skew 500 ppm
I20260810 06:25:28.028093 13593 webserver.cc:460] Webserver started at http://127.4.73.62:39885/ using document root <none> and password file <none>
I20260810 06:25:28.028720 13593 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:28.028807 13593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:28.029045 13593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:28.031262 13593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "9cef66a479254804a67feed9f0221c2f"
format_stamp: "Formatted at 2026-08-10 06:25:28 on dist-test-slave-45dx"
server_key: "f4fcfe43ef2b6cf4122aa09042b0660f"
server_key_iv: "431056431cf3e66951f0e38b5f98ebc5"
server_key_version: "encryptionkey@0"
I20260810 06:25:28.031941 13593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "9cef66a479254804a67feed9f0221c2f"
format_stamp: "Formatted at 2026-08-10 06:25:28 on dist-test-slave-45dx"
server_key: "f4fcfe43ef2b6cf4122aa09042b0660f"
server_key_iv: "431056431cf3e66951f0e38b5f98ebc5"
server_key_version: "encryptionkey@0"
I20260810 06:25:28.035447 13593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:25:28.037832 13618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:28.039008 13593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:28.039156 13593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "9cef66a479254804a67feed9f0221c2f"
format_stamp: "Formatted at 2026-08-10 06:25:28 on dist-test-slave-45dx"
server_key: "f4fcfe43ef2b6cf4122aa09042b0660f"
server_key_iv: "431056431cf3e66951f0e38b5f98ebc5"
server_key_version: "encryptionkey@0"
I20260810 06:25:28.039274 13593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:28.055573 13593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:28.056172 13593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:28.056397 13593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:28.063324 13593 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:34417
I20260810 06:25:28.063333 13704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:34417 every 8 connection(s)
I20260810 06:25:28.064191 13593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:28.067139 13705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:28.070477 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 13593
I20260810 06:25:28.070604 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:28.070855 4388 external_mini_cluster.cc:1428] Setting key ded6d469c50146de38008aba689a4c25
I20260810 06:25:28.072625 13705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f: Bootstrap starting.
I20260810 06:25:28.074812 13705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:28.075594 13705 log.cc:824] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:28.077666 13705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f: No bootstrap required, opened a new log
I20260810 06:25:28.079411 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:28.072123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.4.73.62:34417)
0810 06:25:28.072680 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:28.072694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:28.072725 (+ 31us) server_negotiation.cc:407] Connection header received
0810 06:25:28.073307 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:28.073322 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:28.073666 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:28.073912 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:28.074239 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:28.075199 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:28.076253 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:28.076787 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:28.076918 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:28.077834 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:28.077839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:28.077850 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:25:28.078210 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:28.078261 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:25:28.078281 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:28.078354 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:28.078525 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 06:25:28.078798 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":139,"threads_started":1}
I20260810 06:25:28.081298 13705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cef66a479254804a67feed9f0221c2f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34417 } }
I20260810 06:25:28.081532 13705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:28.081610 13705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cef66a479254804a67feed9f0221c2f, State: Initialized, Role: FOLLOWER
I20260810 06:25:28.082113 13705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cef66a479254804a67feed9f0221c2f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34417 } }
I20260810 06:25:28.082195 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:25:28.082281 13705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:28.082347 13705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:28.082441 13705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:28.083313 13705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cef66a479254804a67feed9f0221c2f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34417 } }
I20260810 06:25:28.083640 13705 leader_election.cc:302] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cef66a479254804a67feed9f0221c2f; no voters:
I20260810 06:25:28.083966 13705 leader_election.cc:288] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:28.084081 13712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:28.084336 13712 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [term 1 LEADER]: Becoming Leader. State: Replica: 9cef66a479254804a67feed9f0221c2f, State: Running, Role: LEADER
I20260810 06:25:28.084612 13712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cef66a479254804a67feed9f0221c2f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34417 } }
I20260810 06:25:28.085122 13705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:28.086262 13716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cef66a479254804a67feed9f0221c2f. Latest consensus state: current_term: 1 leader_uuid: "9cef66a479254804a67feed9f0221c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cef66a479254804a67feed9f0221c2f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34417 } } }
I20260810 06:25:28.086253 13713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cef66a479254804a67feed9f0221c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cef66a479254804a67feed9f0221c2f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34417 } } }
I20260810 06:25:28.086397 13713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:28.086397 13716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:28.086830 13733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:28.089790 13733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:28.094933 13733 catalog_manager.cc:1347] Generated new cluster ID: f10c9b9122fb4f6c9ae93db7b3d1b357
I20260810 06:25:28.095010 13733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:28.110369 13733 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:28.111225 13733 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:28.118062 13733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cef66a479254804a67feed9f0221c2f: Generated new TSK 0
I20260810 06:25:28.118690 13733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:25:28.574702 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:28.128502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.4.73.62:34417)
0810 06:25:28.128639 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:28.128644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:28.128835 (+ 191us) server_negotiation.cc:407] Connection header received
0810 06:25:28.129008 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:28.129010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:28.129054 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:28.129132 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0810 06:25:28.129404 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:28.129936 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:28.131157 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:28.131384 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:28.131459 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:28.131802 (+ 343us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 06:25:28.574293 (+442491us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 06:25:28.574560 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53090: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 06:25:28.574828 13710 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53090: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 06:25:28.574988 13765 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:28.128416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34417 (local address 127.0.0.1:53090)
0810 06:25:28.128741 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:28.128756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:28.128875 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:28.129164 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:28.129167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:28.129175 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 06:25:28.129300 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:28.129306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:28.130068 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:28.130072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:28.131040 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:28.131051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:28.131488 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:28.131523 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:28.131616 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:28.131643 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 06:25:28.574530 (+442887us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 06:25:28.574754 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.73.62:34417: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":122,"threads_started":1}
I20260810 06:25:28.575757 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 13593
2026-08-10T06:25:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (719 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:28 dist-test-slave-45dx krb5kdc[13785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:28 dist-test-slave-45dx krb5kdc[13785](info): set up 2 sockets
Aug 10 06:25:28 dist-test-slave-45dx krb5kdc[13785](info): commencing operation
krb5kdc: starting...
W20260810 06:25:30.073951 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:30 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:30.194522 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44577
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:44577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:30.305073 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:30.305338 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:30.305415 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:30.309387 13809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:30.309468 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:30.309505 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:30.309585 13809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:30.309641 13809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:30.315001 13809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:44577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:30.315975 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:30.317046 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:30.325451 13821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:30.325462 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:30.325462 13816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:30.325527 13809 server_base.cc:1034] running on GCE node
I20260810 06:25:30.326329 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:30.327366 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:30.328552 13809 hybrid_clock.cc:648] HybridClock initialized: now 1786343130328534 us; error 41 us; skew 500 ppm
Aug 10 06:25:30 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343130, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:30.334724 13809 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:30.335632 13809 webserver.cc:460] Webserver started at http://127.4.73.62:38649/ using document root <none> and password file <none>
I20260810 06:25:30.336140 13809 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:30.336200 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:30.336447 13809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:30.338388 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "224bf033c0d3419ab6b58f06e79cb5b5"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "8fe890962902f571bc53aa4083279e53"
server_key_iv: "4616d7997804f170a066a8e74185f7b3"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.338888 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "224bf033c0d3419ab6b58f06e79cb5b5"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "8fe890962902f571bc53aa4083279e53"
server_key_iv: "4616d7997804f170a066a8e74185f7b3"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.342417 13809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:25:30.344688 13832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:30.345873 13809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:30.345997 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "224bf033c0d3419ab6b58f06e79cb5b5"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "8fe890962902f571bc53aa4083279e53"
server_key_iv: "4616d7997804f170a066a8e74185f7b3"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.346093 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:30.356138 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:30.358979 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:30.359180 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:30.366272 13809 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:44577
I20260810 06:25:30.366305 13919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:44577 every 8 connection(s)
I20260810 06:25:30.367189 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:30.369853 13920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:30.370594 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 13809
I20260810 06:25:30.370744 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:30.371057 4388 external_mini_cluster.cc:1428] Setting key a5c2babc0328df5b9679806aa90db479
I20260810 06:25:30.376080 13920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5: Bootstrap starting.
I20260810 06:25:30.378351 13920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:30.379087 13920 log.cc:824] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:25:30 dist-test-slave-45dx krb5kdc[13785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:30.381201 13920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5: No bootstrap required, opened a new log
I20260810 06:25:30.384835 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:30.372551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.4.73.62:44577)
0810 06:25:30.372973 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:30.372987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:30.373021 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:25:30.373662 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:30.373677 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:30.373973 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:30.374319 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:30.375124 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.376177 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.377097 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.377567 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.377743 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:30.380411 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:30.380416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:30.380427 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:30.380452 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:30.382106 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:30.382567 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:30.382572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:30.382577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:30.382683 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:30.383013 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:30.383016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:30.383018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:30.383381 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:30.383580 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:30.383846 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 06:25:30.384115 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":132,"threads_started":1}
I20260810 06:25:30.384864 13920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44577 } }
I20260810 06:25:30.385151 13920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:30.385210 13920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224bf033c0d3419ab6b58f06e79cb5b5, State: Initialized, Role: FOLLOWER
I20260810 06:25:30.385720 13920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44577 } }
I20260810 06:25:30.385845 13920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:30.385931 13920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:30.386029 13920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:30.387055 13920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44577 } }
I20260810 06:25:30.387405 13920 leader_election.cc:302] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224bf033c0d3419ab6b58f06e79cb5b5; no voters:
I20260810 06:25:30.387689 13920 leader_election.cc:288] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:30.387816 13927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:30.388075 13927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [term 1 LEADER]: Becoming Leader. State: Replica: 224bf033c0d3419ab6b58f06e79cb5b5, State: Running, Role: LEADER
I20260810 06:25:30.388440 13927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44577 } }
I20260810 06:25:30.388866 13920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:30.390504 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44577 } } }
I20260810 06:25:30.390636 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:30.390965 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224bf033c0d3419ab6b58f06e79cb5b5. Latest consensus state: current_term: 1 leader_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224bf033c0d3419ab6b58f06e79cb5b5" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44577 } } }
I20260810 06:25:30.391091 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5 [sys.catalog]: This master's current role is: LEADER
W20260810 06:25:30.393436 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:25:30.393515 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:25:30.393572 13957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:30.394256 13957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:30.399012 13957 catalog_manager.cc:1347] Generated new cluster ID: 562f45f7cd0d4b1cafae4b61d2138acd
I20260810 06:25:30.399077 13957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:30.414566 13957 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:30.415524 13957 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:30.423496 13957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224bf033c0d3419ab6b58f06e79cb5b5: Generated new TSK 0
I20260810 06:25:30.424218 13957 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:30.464567 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44577
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:30.571628 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:30.571847 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:30.571885 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:30.575716 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:30.575765 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:30.575817 13968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:30.580731 13968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:30.581621 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:30.582544 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:30.591889 13977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:30.591878 13978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:30.591878 13980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:30.592032 13968 server_base.cc:1034] running on GCE node
I20260810 06:25:30.592612 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:30.593215 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:30.594403 13968 hybrid_clock.cc:648] HybridClock initialized: now 1786343130594388 us; error 34 us; skew 500 ppm
Aug 10 06:25:30 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343130, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:30.597211 13968 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:30.597968 13968 webserver.cc:460] Webserver started at http://127.4.73.1:36963/ using document root <none> and password file <none>
I20260810 06:25:30.598456 13968 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:30.598520 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:30.598672 13968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:30.600625 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "e476255768d84a5391f6d300fb6f1584"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "e3b804d8bebbf26f7e3375d1008066ae"
server_key_iv: "d7ab726cae8e58acda32de299dd763d8"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.601146 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "e476255768d84a5391f6d300fb6f1584"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "e3b804d8bebbf26f7e3375d1008066ae"
server_key_iv: "d7ab726cae8e58acda32de299dd763d8"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.604666 13968 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 06:25:30.606922 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:30.608044 13968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:30.608139 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "e476255768d84a5391f6d300fb6f1584"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "e3b804d8bebbf26f7e3375d1008066ae"
server_key_iv: "d7ab726cae8e58acda32de299dd763d8"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.608218 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:30.622936 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:30.625981 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:30.626120 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:30.626583 13968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:30.627321 13968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:30.627367 13968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:30.627404 13968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:30.627425 13968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:30.636123 13968 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:41513
I20260810 06:25:30.636145 14173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:41513 every 8 connection(s)
I20260810 06:25:30.637028 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:30.640443 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 13968
I20260810 06:25:30.640576 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:30.640872 4388 external_mini_cluster.cc:1428] Setting key c9922ef29491d84554195ffb2aaa4c84
Aug 10 06:25:30 dist-test-slave-45dx krb5kdc[13785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343130, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:30.651898 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:30.638816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:35561 (local address 127.4.73.62:44577)
0810 06:25:30.638982 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:30.638987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:30.639794 (+ 807us) server_negotiation.cc:407] Connection header received
0810 06:25:30.641412 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:30.641415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:30.641477 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:30.641592 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:30.642727 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.643252 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.644711 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.644940 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.645031 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:30.648019 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:30.648027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:30.648032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:30.648083 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:30.649694 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:30.650255 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:30.650261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:30.650264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:30.650406 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:30.650779 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:30.650784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:30.650786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:30.650973 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:30.651085 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:30.651576 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 06:25:30.651740 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:25:30.652771 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:30.639183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44577 (local address 127.4.73.1:35561)
0810 06:25:30.639604 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:30.639656 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:30.641197 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:30.641769 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:30.641776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:30.642126 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:30.642575 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:30.642588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.643377 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.643381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:30.644583 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:30.644598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.645103 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.645107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:30.645330 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:30.645959 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:30.645975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:30.647786 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:30.649839 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:30.649844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:30.649857 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:30.650126 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:30.650529 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:30.650532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:30.650535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:30.650679 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:30.651086 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:30.651093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:30.651321 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 06:25:30.651525 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 06:25:30.651903 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260810 06:25:30.653960 14177 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44577
I20260810 06:25:30.654181 14177 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:30.654716 14177 heartbeater.cc:511] Master 127.4.73.62:44577 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:30.656418 13861 ts_manager.cc:194] Registered new tserver with Master: e476255768d84a5391f6d300fb6f1584 (127.4.73.1:41513)
I20260810 06:25:30.657635 13861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:35561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:30.669414 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44577
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:30.778798 14188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:30.779069 14188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:30.779136 14188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:30.783104 14188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:30.783160 14188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:30.783376 14188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:30.788950 14188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:30.790061 14188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:30.791086 14188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:30.800895 14196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:30.800943 14199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:30.801097 14188 server_base.cc:1034] running on GCE node
W20260810 06:25:30.800895 14197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:30.801601 14188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:30.802170 14188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:30.803328 14188 hybrid_clock.cc:648] HybridClock initialized: now 1786343130803296 us; error 44 us; skew 500 ppm
Aug 10 06:25:30 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343130, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:30.806185 14188 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:30.807042 14188 webserver.cc:460] Webserver started at http://127.4.73.2:39069/ using document root <none> and password file <none>
I20260810 06:25:30.807596 14188 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:30.807658 14188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:30.807924 14188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:30.809935 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "4d4f78af296b49bc8bbc5e78c2249ee1"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "11c9712202012ad34bfcb0876740a878"
server_key_iv: "861fbbe57280cf8fb5856b6331c56457"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.810505 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "4d4f78af296b49bc8bbc5e78c2249ee1"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "11c9712202012ad34bfcb0876740a878"
server_key_iv: "861fbbe57280cf8fb5856b6331c56457"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.814148 14188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:25:30.816515 14214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:30.817699 14188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:30.817821 14188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "4d4f78af296b49bc8bbc5e78c2249ee1"
format_stamp: "Formatted at 2026-08-10 06:25:30 on dist-test-slave-45dx"
server_key: "11c9712202012ad34bfcb0876740a878"
server_key_iv: "861fbbe57280cf8fb5856b6331c56457"
server_key_version: "encryptionkey@0"
I20260810 06:25:30.817924 14188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:30.842245 14188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:30.845227 14188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:30.845455 14188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:30.846021 14188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:30.846864 14188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:30.846911 14188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:30.846983 14188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:30.847021 14188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:30.855667 14188 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:46399
I20260810 06:25:30.855736 14386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:46399 every 8 connection(s)
I20260810 06:25:30.856585 14188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
Aug 10 06:25:30 dist-test-slave-45dx krb5kdc[13785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343130, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:30.865805 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 14188
I20260810 06:25:30.865949 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:30.866273 4388 external_mini_cluster.cc:1428] Setting key 3be35b08282b00f961d69aad4d6a8252
I20260810 06:25:30.869908 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:30.858345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:42639 (local address 127.4.73.62:44577)
0810 06:25:30.858518 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:30.858523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:30.859291 (+ 768us) server_negotiation.cc:407] Connection header received
0810 06:25:30.860613 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:30.860616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:30.860662 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:30.860734 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:30.861937 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.862462 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.863544 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.863746 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.863818 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:30.866258 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:30.866263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:30.866265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:30.866294 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:30.867861 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:30.868394 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:30.868397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:30.868399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:30.868513 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:30.868864 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:30.868867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:30.868869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:30.869043 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:30.869145 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:30.869629 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 06:25:30.869746 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:25:30.870644 14392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:30.858706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44577 (local address 127.4.73.2:42639)
0810 06:25:30.859144 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:30.859185 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:30.860432 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:30.860887 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:30.860893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:30.861337 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:30.861814 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:30.861828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.862575 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.862577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:30.863444 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:30.863453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:30.863867 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:30.863870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:30.864014 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:30.864610 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:30.864627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:30.866092 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:30.867993 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:30.867998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:30.868022 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:30.868267 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:30.868608 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:30.868614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:30.868617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:30.868763 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:30.869144 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:30.869152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:30.869371 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 06:25:30.869574 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 06:25:30.869877 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260810 06:25:30.871755 14388 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44577
I20260810 06:25:30.872008 14388 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:30.872591 14388 heartbeater.cc:511] Master 127.4.73.62:44577 requested a full tablet report, sending...
I20260810 06:25:30.873759 13861 ts_manager.cc:194] Registered new tserver with Master: 4d4f78af296b49bc8bbc5e78c2249ee1 (127.4.73.2:46399)
I20260810 06:25:30.874650 13861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:42639
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:30.893366 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44577
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:31.002250 14395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:31.002506 14395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:31.002547 14395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:31.006404 14395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:31.006449 14395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:31.006501 14395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:31.011495 14395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:31.012426 14395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:31.013375 14395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:31.022670 14402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:31.022683 14404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:31.022742 14395 server_base.cc:1034] running on GCE node
W20260810 06:25:31.022742 14406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:31.023406 14395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:31.024024 14395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:31.025215 14395 hybrid_clock.cc:648] HybridClock initialized: now 1786343131025179 us; error 65 us; skew 500 ppm
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.027941 14395 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:31.028890 14395 webserver.cc:460] Webserver started at http://127.4.73.3:33543/ using document root <none> and password file <none>
I20260810 06:25:31.029479 14395 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:31.029597 14395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:31.029837 14395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:31.031836 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "ef51ea390f2843eb8da752956025131b"
format_stamp: "Formatted at 2026-08-10 06:25:31 on dist-test-slave-45dx"
server_key: "f4bf5c2998942d9da85e3314180a66d3"
server_key_iv: "38edcf53254c719a909ae278c5d09dd0"
server_key_version: "encryptionkey@0"
I20260810 06:25:31.032452 14395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "ef51ea390f2843eb8da752956025131b"
format_stamp: "Formatted at 2026-08-10 06:25:31 on dist-test-slave-45dx"
server_key: "f4bf5c2998942d9da85e3314180a66d3"
server_key_iv: "38edcf53254c719a909ae278c5d09dd0"
server_key_version: "encryptionkey@0"
I20260810 06:25:31.036010 14395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:31.038396 14416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:31.039608 14395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:25:31.039744 14395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "ef51ea390f2843eb8da752956025131b"
format_stamp: "Formatted at 2026-08-10 06:25:31 on dist-test-slave-45dx"
server_key: "f4bf5c2998942d9da85e3314180a66d3"
server_key_iv: "38edcf53254c719a909ae278c5d09dd0"
server_key_version: "encryptionkey@0"
I20260810 06:25:31.039862 14395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:31.065400 14395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:31.068363 14395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:31.068567 14395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:31.069180 14395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:31.069975 14395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:31.070050 14395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:31.070122 14395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:31.070165 14395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:31.078985 14395 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:44373
I20260810 06:25:31.079021 14607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:44373 every 8 connection(s)
I20260810 06:25:31.079839 14395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:31.089548 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 14395
I20260810 06:25:31.089701 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:31.090034 4388 external_mini_cluster.cc:1428] Setting key de957603b2be07b78274193e32204cf9
I20260810 06:25:31.093811 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:31.081612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:60333 (local address 127.4.73.62:44577)
0810 06:25:31.081753 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:31.081757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:31.082519 (+ 762us) server_negotiation.cc:407] Connection header received
0810 06:25:31.083828 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:31.083831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:31.083881 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:31.083956 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:31.085138 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.085653 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.086923 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.087140 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.087224 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:31.089761 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:31.089768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:31.089773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:31.089813 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:31.091736 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.092389 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.092394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.092399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.092503 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.092876 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.092879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.092880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.093060 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:31.093134 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:31.093601 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 06:25:31.093698 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:25:31.094581 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:31.081903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44577 (local address 127.4.73.3:60333)
0810 06:25:31.082354 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:31.082395 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:31.083636 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:31.084116 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:31.084122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:31.084547 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:31.084999 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:31.085012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.085756 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.085758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:31.086795 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:31.086804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.087267 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.087269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:31.087407 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:31.087916 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:31.087932 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:31.089552 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:31.091847 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:31.091852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:31.091866 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:31.092245 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:31.092614 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:31.092617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:31.092619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:31.092781 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:31.093149 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:31.093154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:31.093377 (+ 223us) client_negotiation.cc:769] Sending connection context
0810 06:25:31.093521 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 06:25:31.093762 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260810 06:25:31.095676 14608 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44577
I20260810 06:25:31.095919 14608 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:31.096431 14608 heartbeater.cc:511] Master 127.4.73.62:44577 requested a full tablet report, sending...
I20260810 06:25:31.097409 13861 ts_manager.cc:194] Registered new tserver with Master: ef51ea390f2843eb8da752956025131b (127.4.73.3:44373)
I20260810 06:25:31.097934 13861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:60333
I20260810 06:25:31.104283 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/test-admin.keytab.
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.120875 4388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 06:25:31.123695 14618 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:31.122257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.4.73.1:41513)
0810 06:25:31.122643 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:31.122660 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:31.122687 (+ 27us) server_negotiation.cc:407] Connection header received
0810 06:25:31.122746 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:31.122760 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:31.122938 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:31.123069 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:31.123517 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52556: BlockingRecv error: recv got EOF from 127.0.0.1:52556 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
W20260810 06:25:31.123734 4388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5cc' of type FILE
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.125000 4388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.125886 4388 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:31.135001 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:31.126711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.4.73.1:41513)
0810 06:25:31.126876 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:31.126881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:31.126955 (+ 74us) server_negotiation.cc:407] Connection header received
0810 06:25:31.127104 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:31.127112 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:31.127151 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:31.127234 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:31.127763 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.128601 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.129403 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.129754 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.129875 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:31.132060 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:31.132069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:31.132080 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:31.132102 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:31.133550 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.133929 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.133934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.133935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.134008 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.134227 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.134229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.134230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.134437 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:31.134584 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:31.134758 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 06:25:31.134873 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 06:25:31.138550 14618 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:31.137686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.4.73.1:41513)
0810 06:25:31.137817 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:31.137820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:31.137835 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:25:31.137947 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:31.137949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:31.137990 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:31.138052 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:31.138456 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52568: BlockingRecv error: recv got EOF from 127.0.0.1:52568 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 06:25:31.138665 4388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5cc' of type FILE
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.139668 4388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.140560 4388 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:31.148635 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:31.141335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.4.73.1:41513)
0810 06:25:31.141441 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:31.141444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:31.141515 (+ 71us) server_negotiation.cc:407] Connection header received
0810 06:25:31.141590 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:31.141592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:31.141623 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:31.141684 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:31.142131 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.142642 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.143343 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.143614 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.143691 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:31.145793 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:31.145796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:31.145798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:31.145819 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:31.147229 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.147575 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.147578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.147579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.147650 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.147867 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.147869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.147870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.148019 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:31.148121 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:31.148357 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 06:25:31.148478 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 06:25:31.151263 14618 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:31.150295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.4.73.1:41513)
0810 06:25:31.150474 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:31.150481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:31.150495 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:31.150631 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:31.150634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:31.150678 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:31.150760 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:31.151154 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52580: BlockingRecv error: recv got EOF from 127.0.0.1:52580 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 06:25:31.151410 4388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786343096561320-4388-0/krb5kdc/krb5cc' of type FILE
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.152333 4388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:31.153244 4388 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[13785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:25:31.161269 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:31.153888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.4.73.1:41513)
0810 06:25:31.154008 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:31.154013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:31.154127 (+ 114us) server_negotiation.cc:407] Connection header received
0810 06:25:31.154203 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:31.154204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:31.154238 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:31.154291 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:31.154720 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.155180 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.156123 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:31.156364 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:31.156455 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:31.158430 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:31.158433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:31.158435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:31.158455 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:31.159829 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.160186 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.160188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.160189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.160257 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:31.160539 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:31.160541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:31.160542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:31.160678 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:31.160786 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:31.161006 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 06:25:31.161133 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:25:31.162185 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 13968
I20260810 06:25:31.178274 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 14188
I20260810 06:25:31.194876 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 14395
I20260810 06:25:31.203807 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 13809
2026-08-10T06:25:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2378 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[14637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[14637](info): set up 2 sockets
Aug 10 06:25:31 dist-test-slave-45dx krb5kdc[14637](info): commencing operation
krb5kdc: starting...
W20260810 06:25:32.461156 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:32 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab.
Entry for principal oryx/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab.
I20260810 06:25:32.580281 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:39809
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab
--principal=oryx/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:39809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:32.687414 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:32.687667 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:32.687704 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:32.691493 14656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:32.691542 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:32.691561 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:32.691577 14656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:32.691592 14656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:32.696538 14656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:39809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab
--principal=oryx/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:39809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:32.697428 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:32.698335 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:32.706661 14664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:32.706710 14656 server_base.cc:1034] running on GCE node
W20260810 06:25:32.706741 14661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:32.706691 14667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:32.707468 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:32.708534 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:32.709702 14656 hybrid_clock.cc:648] HybridClock initialized: now 1786343132709682 us; error 31 us; skew 500 ppm
Aug 10 06:25:32 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:32.712411 14656 init.cc:377] Logged in from keytab as oryx/127.4.73.62@KRBTEST.COM (short username oryx)
I20260810 06:25:32.713255 14656 webserver.cc:460] Webserver started at http://127.4.73.62:33535/ using document root <none> and password file <none>
I20260810 06:25:32.713841 14656 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:32.713937 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:32.714165 14656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:32.716149 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "45105579b0a74e16b191d3ad16b5a7b7"
format_stamp: "Formatted at 2026-08-10 06:25:32 on dist-test-slave-45dx"
server_key: "209fdb6b918ccbd13d1504e4732e549c"
server_key_iv: "1084b055e7460be53b64dc602ea27983"
server_key_version: "encryptionkey@0"
I20260810 06:25:32.716779 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "45105579b0a74e16b191d3ad16b5a7b7"
format_stamp: "Formatted at 2026-08-10 06:25:32 on dist-test-slave-45dx"
server_key: "209fdb6b918ccbd13d1504e4732e549c"
server_key_iv: "1084b055e7460be53b64dc602ea27983"
server_key_version: "encryptionkey@0"
I20260810 06:25:32.720405 14656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:25:32.722822 14682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:32.724016 14656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:25:32.724156 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "45105579b0a74e16b191d3ad16b5a7b7"
format_stamp: "Formatted at 2026-08-10 06:25:32 on dist-test-slave-45dx"
server_key: "209fdb6b918ccbd13d1504e4732e549c"
server_key_iv: "1084b055e7460be53b64dc602ea27983"
server_key_version: "encryptionkey@0"
I20260810 06:25:32.724267 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:32.744889 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:32.747529 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:32.747725 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:32.754470 14656 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:39809
I20260810 06:25:32.754762 14760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:39809 every 8 connection(s)
I20260810 06:25:32.755412 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:32.758257 14761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:32.763613 14761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: Bootstrap starting.
I20260810 06:25:32.765583 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 14656
I20260810 06:25:32.765733 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:32.766021 4388 external_mini_cluster.cc:1428] Setting key 0ab5f141bba6e1fb173f2ece59047eb6
I20260810 06:25:32.766172 14761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:32.767122 14761 log.cc:824] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:32.769133 14761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: No bootstrap required, opened a new log
I20260810 06:25:32.773134 14761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } }
I20260810 06:25:32.773406 14761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:32.773473 14761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45105579b0a74e16b191d3ad16b5a7b7, State: Initialized, Role: FOLLOWER
I20260810 06:25:32.773995 14761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } }
I20260810 06:25:32.774168 14761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:32.774247 14761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:32.774367 14761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 06:25:32 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:32.775295 14761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } }
I20260810 06:25:32.775617 14761 leader_election.cc:302] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45105579b0a74e16b191d3ad16b5a7b7; no voters:
I20260810 06:25:32.775923 14761 leader_election.cc:288] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:32.776170 14766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:32.776788 14766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 1 LEADER]: Becoming Leader. State: Replica: 45105579b0a74e16b191d3ad16b5a7b7, State: Running, Role: LEADER
I20260810 06:25:32.777132 14766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } }
I20260810 06:25:32.777252 14761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:32.778276 14768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45105579b0a74e16b191d3ad16b5a7b7. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } }
I20260810 06:25:32.778411 14768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:32.778790 14767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } }
I20260810 06:25:32.778970 14767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:32.780570 14773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:32.781564 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:32.767210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.4.73.62:39809)
0810 06:25:32.767708 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:32.767723 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:32.767758 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:25:32.768449 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:32.768467 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:32.768782 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:32.769065 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:32.769966 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:32.770908 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:32.772112 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:32.772616 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:32.772736 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:32.775446 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:32.775453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:32.775469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:32.775507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:32.777720 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:32.778230 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:32.778236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:32.778241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:32.778395 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:32.778750 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:32.778755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:32.778758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:32.779319 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:32.779552 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:32.780497 (+ 945us) server_negotiation.cc:299] Negotiation successful
0810 06:25:32.780841 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":181,"threads_started":1}
I20260810 06:25:32.783510 14773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:32.788426 14773 catalog_manager.cc:1347] Generated new cluster ID: a798e26e8a0e4b7ea5f122b7386bd3fc
I20260810 06:25:32.788498 14773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:32.799853 14773 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:32.801020 14773 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:32.808614 14773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: Generated new TSK 0
I20260810 06:25:32.809273 14773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab.
Entry for principal oryx/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab.
I20260810 06:25:32.842679 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39809
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab
--principal=oryx/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:32.952530 14805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:32.952771 14805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:32.952829 14805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:32.956671 14805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:32.956718 14805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:32.956822 14805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:32.961836 14805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab
--principal=oryx/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:32.962764 14805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:32.963719 14805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:32.973060 14812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:32.973044 14814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:32.973044 14816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:32.973253 14805 server_base.cc:1034] running on GCE node
I20260810 06:25:32.973744 14805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:32.974375 14805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:32.975569 14805 hybrid_clock.cc:648] HybridClock initialized: now 1786343132975558 us; error 29 us; skew 500 ppm
Aug 10 06:25:32 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:32.978364 14805 init.cc:377] Logged in from keytab as oryx/127.4.73.1@KRBTEST.COM (short username oryx)
I20260810 06:25:32.979380 14805 webserver.cc:460] Webserver started at http://127.4.73.1:44267/ using document root <none> and password file <none>
I20260810 06:25:32.979916 14805 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:32.980012 14805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:32.980283 14805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:32.982270 14805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "750304f89b244671bb6de42e1cfa4c84"
format_stamp: "Formatted at 2026-08-10 06:25:32 on dist-test-slave-45dx"
server_key: "2ce944e0bfbc054d1777660a7bc695ae"
server_key_iv: "f8b02f9004dfd0d0530ce206023cc9be"
server_key_version: "encryptionkey@0"
I20260810 06:25:32.982836 14805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "750304f89b244671bb6de42e1cfa4c84"
format_stamp: "Formatted at 2026-08-10 06:25:32 on dist-test-slave-45dx"
server_key: "2ce944e0bfbc054d1777660a7bc695ae"
server_key_iv: "f8b02f9004dfd0d0530ce206023cc9be"
server_key_version: "encryptionkey@0"
I20260810 06:25:32.986446 14805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:25:32.988847 14824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:32.989995 14805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:32.990115 14805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "750304f89b244671bb6de42e1cfa4c84"
format_stamp: "Formatted at 2026-08-10 06:25:32 on dist-test-slave-45dx"
server_key: "2ce944e0bfbc054d1777660a7bc695ae"
server_key_iv: "f8b02f9004dfd0d0530ce206023cc9be"
server_key_version: "encryptionkey@0"
I20260810 06:25:32.990212 14805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:33.020936 14805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:33.023672 14805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:33.023880 14805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:33.024490 14805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:33.025305 14805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:33.025353 14805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.025422 14805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:33.025452 14805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.034279 14805 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:44507
I20260810 06:25:33.034309 15002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:44507 every 8 connection(s)
I20260810 06:25:33.035149 14805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:33.039206 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 14805
I20260810 06:25:33.039374 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:33.039654 4388 external_mini_cluster.cc:1428] Setting key 06c36eca95962f673d5d4c2051ecbf84
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:33.049921 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.036923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:43555 (local address 127.4.73.62:39809)
0810 06:25:33.037070 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.037074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.037823 (+ 749us) server_negotiation.cc:407] Connection header received
0810 06:25:33.039110 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.039113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.039191 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.039310 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.040751 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.041515 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.042695 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.043126 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.043245 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.046095 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.046100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.046105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.046138 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.047708 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.048249 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.048253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.048255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.048419 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.048804 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.048808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.048809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.049014 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.049124 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.049612 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.049743 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for oryx/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:25:33.050761 15006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.037245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:43555)
0810 06:25:33.037676 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.037718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.038901 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.039519 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.039527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.039932 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.040607 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.040625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.041657 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.041660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.042564 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.042574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.043301 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.043307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.043541 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.044176 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.044199 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.045895 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.047842 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.047848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.047862 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.048138 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.048536 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.048539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.048541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.048699 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.049125 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.049133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.049375 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.049566 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.049887 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":121,"threads_started":1}
I20260810 06:25:33.051967 15004 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39809
I20260810 06:25:33.052237 15004 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:33.052862 15004 heartbeater.cc:511] Master 127.4.73.62:39809 requested a full tablet report, sending...
I20260810 06:25:33.054447 14705 ts_manager.cc:194] Registered new tserver with Master: 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507)
I20260810 06:25:33.055531 14705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:43555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab.
Entry for principal oryx/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab.
I20260810 06:25:33.067699 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39809
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab
--principal=oryx/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:33.174682 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:33.174907 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:33.174947 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:33.178926 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:33.178977 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:33.179030 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:33.184003 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab
--principal=oryx/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:33.184928 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:33.185799 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:33.195312 15022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.195381 15011 server_base.cc:1034] running on GCE node
W20260810 06:25:33.195322 15018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:33.195292 15020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.195945 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:33.196597 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:33.197759 15011 hybrid_clock.cc:648] HybridClock initialized: now 1786343133197736 us; error 35 us; skew 500 ppm
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:33.200512 15011 init.cc:377] Logged in from keytab as oryx/127.4.73.2@KRBTEST.COM (short username oryx)
I20260810 06:25:33.201275 15011 webserver.cc:460] Webserver started at http://127.4.73.2:42149/ using document root <none> and password file <none>
I20260810 06:25:33.201792 15011 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:33.201854 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:33.202013 15011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:33.203904 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "1445e6450e8c4466a7f82873e98327a4"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "8f7565aab5cddc2672c656f9edb26454"
server_key_iv: "9fe2eb3e92936523672601d3efe07eb9"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.204496 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "1445e6450e8c4466a7f82873e98327a4"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "8f7565aab5cddc2672c656f9edb26454"
server_key_iv: "9fe2eb3e92936523672601d3efe07eb9"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.207986 15011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:33.210309 15031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.211545 15011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:33.211688 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "1445e6450e8c4466a7f82873e98327a4"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "8f7565aab5cddc2672c656f9edb26454"
server_key_iv: "9fe2eb3e92936523672601d3efe07eb9"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.211798 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:33.226621 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:33.229512 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:33.229702 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:33.230247 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:33.231053 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:33.231133 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.231199 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:33.231235 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.239576 15011 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:35799
I20260810 06:25:33.239614 15216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:35799 every 8 connection(s)
I20260810 06:25:33.240440 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:33.243497 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 15011
I20260810 06:25:33.243638 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:33.243935 4388 external_mini_cluster.cc:1428] Setting key a55f4f809fe7f60c58ec7cd3c7984e7e
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:33.253845 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.242035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:37727 (local address 127.4.73.62:39809)
0810 06:25:33.242198 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.242203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.242956 (+ 753us) server_negotiation.cc:407] Connection header received
0810 06:25:33.244340 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.244344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.244408 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.244527 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.245698 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.246231 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.247372 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.247600 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.247714 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.250059 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.250065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.250071 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.250104 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.251729 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.252202 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.252208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.252212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.252343 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.252679 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.252685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.252688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.252938 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.253045 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.253499 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.253640 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 06:25:33.254442 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.242392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:37727)
0810 06:25:33.242808 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.242847 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.244113 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.244703 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.244718 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.245085 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.245565 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.245577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.246381 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.246383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.247242 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.247252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.247751 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.247756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.247923 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.248531 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.248546 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.249907 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.251842 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.251847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.251856 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.252102 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.252445 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.252447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.252448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.252584 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.253041 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.253046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.253266 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.253409 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.253651 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:25:33.255686 15219 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39809
I20260810 06:25:33.255890 15219 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:33.256395 15219 heartbeater.cc:511] Master 127.4.73.62:39809 requested a full tablet report, sending...
I20260810 06:25:33.257406 14705 ts_manager.cc:194] Registered new tserver with Master: 1445e6450e8c4466a7f82873e98327a4 (127.4.73.2:35799)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.3@KRBTEST.COM" created.
I20260810 06:25:33.258018 14705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:37727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab.
Entry for principal oryx/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab.
I20260810 06:25:33.272662 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39809
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab
--principal=oryx/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:33.385751 15228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:33.385983 15228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:33.386022 15228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:33.389889 15228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:33.389938 15228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:33.390043 15228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:33.395546 15228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab
--principal=oryx/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:33.396582 15228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:33.397562 15228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:33.407347 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:33.407352 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:33.407356 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.407620 15228 server_base.cc:1034] running on GCE node
I20260810 06:25:33.408084 15228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:33.408763 15228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:33.409976 15228 hybrid_clock.cc:648] HybridClock initialized: now 1786343133409960 us; error 24 us; skew 500 ppm
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:33.412925 15228 init.cc:377] Logged in from keytab as oryx/127.4.73.3@KRBTEST.COM (short username oryx)
I20260810 06:25:33.413857 15228 webserver.cc:460] Webserver started at http://127.4.73.3:43045/ using document root <none> and password file <none>
I20260810 06:25:33.414458 15228 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:33.414551 15228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:33.414786 15228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:33.416815 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "86ff81b122d944febf645f52d50d283f"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "9f3141dc21fa64a38e30e337be3ac19a"
server_key_iv: "2680eb031b0b011dbd14f24e642f30e8"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.417397 15228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "86ff81b122d944febf645f52d50d283f"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "9f3141dc21fa64a38e30e337be3ac19a"
server_key_iv: "2680eb031b0b011dbd14f24e642f30e8"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.420995 15228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:33.423307 15249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.424580 15228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:33.424710 15228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "86ff81b122d944febf645f52d50d283f"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "9f3141dc21fa64a38e30e337be3ac19a"
server_key_iv: "2680eb031b0b011dbd14f24e642f30e8"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.424816 15228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:33.436220 15228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:33.438843 15228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:33.439042 15228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:33.439584 15228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:33.440433 15228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:33.440506 15228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.440577 15228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:33.440627 15228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.449793 15228 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:35931
I20260810 06:25:33.449824 15457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:35931 every 8 connection(s)
I20260810 06:25:33.450613 15228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:33.459046 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 15228
I20260810 06:25:33.459167 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:33.459508 4388 external_mini_cluster.cc:1428] Setting key b51b6bf60bd04e89a41ac91d9410ebb0
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:33.464038 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.452353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:36041 (local address 127.4.73.62:39809)
0810 06:25:33.452516 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.452521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.453284 (+ 763us) server_negotiation.cc:407] Connection header received
0810 06:25:33.454518 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.454521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.454568 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.454641 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.455693 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.456195 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.457329 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.457549 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.457627 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.460270 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.460273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.460276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.460360 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.461990 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.462533 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.462538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.462540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.462627 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.462977 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.462982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.462984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.463172 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.463282 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.463759 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.463876 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:25:33.464852 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.452718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:36041)
0810 06:25:33.453134 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.453173 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.454325 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.454793 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.454799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.455120 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.455568 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.455581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.456360 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.456366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.457225 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.457234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.457675 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.457678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.457809 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.458384 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.458400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.460092 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.462122 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.462127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.462140 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.462420 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.462723 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.462726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.462728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.462875 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.463283 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.463290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.463533 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.463714 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.464025 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260810 06:25:33.465926 15458 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39809
I20260810 06:25:33.466117 15458 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:33.466557 15458 heartbeater.cc:511] Master 127.4.73.62:39809 requested a full tablet report, sending...
I20260810 06:25:33.467469 14705 ts_manager.cc:194] Registered new tserver with Master: 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931)
I20260810 06:25:33.467976 14705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:36041
I20260810 06:25:33.473208 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.73.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.61.keytab.
Entry for principal oryx/127.4.73.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.61.keytab.
I20260810 06:25:33.499255 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:33575
--webserver_interface=127.4.73.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.61.keytab
--principal=oryx/127.4.73.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:39809,127.4.73.61:33575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:33.608734 15467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:33.609001 15467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:33.609068 15467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:33.613060 15467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:33.613137 15467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:33.613174 15467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:33.613232 15467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:33.613272 15467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:33.618584 15467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:39809,127.4.73.61:33575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.61.keytab
--principal=oryx/127.4.73.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:33575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.73.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
W20260810 06:25:33.618839 15467 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.73.62:39809,127.4.73.61:33575'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 06:25:33.619643 15467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:33.620728 15467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:33.629189 15483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.629253 15467 server_base.cc:1034] running on GCE node
W20260810 06:25:33.629212 15478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:33.629204 15477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.630024 15467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:33.631098 15467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:33.632284 15467 hybrid_clock.cc:648] HybridClock initialized: now 1786343133632263 us; error 33 us; skew 500 ppm
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:33.635241 15467 init.cc:377] Logged in from keytab as oryx/127.4.73.61@KRBTEST.COM (short username oryx)
I20260810 06:25:33.636050 15467 webserver.cc:460] Webserver started at http://127.4.73.61:42015/ using document root <none> and password file <none>
I20260810 06:25:33.636639 15467 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:33.636703 15467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:33.636863 15467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:33.638792 15467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/instance:
uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "69db13f424eec6ed2b86947ef79c0ce2"
server_key_iv: "825314aa94b637f99dd0834ec4be8b1b"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.639292 15467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal/instance:
uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "69db13f424eec6ed2b86947ef79c0ce2"
server_key_iv: "825314aa94b637f99dd0834ec4be8b1b"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.642876 15467 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 06:25:33.645205 15501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.646358 15467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:33.646461 15467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "69db13f424eec6ed2b86947ef79c0ce2"
server_key_iv: "825314aa94b637f99dd0834ec4be8b1b"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.646569 15467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:33.658914 15467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:33.662025 15467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:33.662171 15467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:33.668828 15467 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.61:33575
I20260810 06:25:33.669116 15590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.61:33575 every 8 connection(s)
I20260810 06:25:33.669764 15467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
I20260810 06:25:33.672426 15592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:33.674839 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 15467
I20260810 06:25:33.676218 15592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.61@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.73.61@KRBTEST.COM
I20260810 06:25:33.688225 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.677950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.4.73.62:39809)
0810 06:25:33.678084 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.678089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.678357 (+ 268us) server_negotiation.cc:407] Connection header received
0810 06:25:33.678584 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.678588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.678634 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.678686 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.679709 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.680229 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.681331 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.681545 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.681626 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.684112 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.684116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.684118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.684140 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.685947 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.686533 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.686539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.686544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.686633 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.687021 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.687024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.687026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.687322 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.687433 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.687934 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.688069 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:33.689018 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.676332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.4.73.61:33575)
0810 06:25:33.676772 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.676788 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.676824 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:25:33.677444 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.677459 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.677808 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.678334 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.679122 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.680293 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.681128 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.681569 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.681753 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.684653 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.684659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.684671 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.684700 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.686505 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.687013 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.687019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.687020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.687134 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.687413 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.687416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.687418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.687662 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.687800 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.688097 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.688222 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":264,"spinlock_wait_cycles":3968,"thread_start_us":121,"threads_started":1}
I20260810 06:25:33.689016 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.677815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:50878)
0810 06:25:33.678247 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.678295 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.678435 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.678857 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.678861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.679306 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.679490 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.679503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.680385 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.680390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.681208 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.681217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.681692 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.681696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.681908 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.682517 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.682533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.683960 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.686055 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.686063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.686070 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.686435 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.686736 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.686739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.686741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.686918 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.687455 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.687460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.687673 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.687856 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.688173 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":39,"thread_start_us":86,"threads_started":1}
I20260810 06:25:33.691013 15592 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.61@KRBTEST.COM for oryx/127.4.73.61@KRBTEST.COM
I20260810 06:25:33.698689 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.691464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.0.0.1:55202)
0810 06:25:33.691572 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.691586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.691672 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.691883 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.691886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.692029 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.692102 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.692106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.692871 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.692874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.693428 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.693437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.694336 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.694339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.694439 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.694787 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.694797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.695907 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.697567 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.697570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.697571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.697745 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.697978 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.697980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.697981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.698051 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.698350 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.698353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.698421 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.698507 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.698593 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 06:25:33.698767 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.691568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55202 (local address 127.4.73.61:33575)
0810 06:25:33.691658 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.691661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.691673 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:25:33.691750 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.691754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.691791 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.691875 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.692207 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.692742 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.693538 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.694221 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.694356 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.696033 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.696036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.696038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.696058 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.697473 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.697827 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.697829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.697831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.697901 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.698126 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.698128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.698129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.698273 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.698344 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.698566 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.698647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 06:25:33.701426 15592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Bootstrap starting.
I20260810 06:25:33.703675 15592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:33.704506 15592 log.cc:824] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:33.706303 15592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: No bootstrap required, opened a new log
I20260810 06:25:33.709260 15592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } }
I20260810 06:25:33.709442 15592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:33.709537 15592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a46bcc18584f2a8b0ba2aa73c5ea26, State: Initialized, Role: FOLLOWER
I20260810 06:25:33.709954 15592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } }
I20260810 06:25:33.710397 15602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } } }
I20260810 06:25:33.710503 15602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:25:33.711103 15592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 06:25:33.716284 15618 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:25:33.716388 15618 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:25:33.718973 15467 master_runner.cc:428] Detected that this master 50a46bcc18584f2a8b0ba2aa73c5ea26 is joining an existing cluster
I20260810 06:25:33.719043 15467 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.73.61:33575
I20260810 06:25:33.731211 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.721646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.0.0.1:55218)
0810 06:25:33.722011 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.722025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.722127 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.722395 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.722399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.722720 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.722895 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.722904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.723871 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.723875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.724755 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.724767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.725373 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.725377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.725472 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.726052 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.726064 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.726478 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.729562 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.729564 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.729566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.729759 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.730064 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.730071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.730073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.730145 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.730599 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.730602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.730735 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.730846 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.730989 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":16,"spinlock_wait_cycles":39168,"thread_start_us":95,"threads_started":1}
I20260810 06:25:33.731269 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.721752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.4.73.61:33575)
0810 06:25:33.721876 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.721880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.722091 (+ 211us) server_negotiation.cc:407] Connection header received
0810 06:25:33.722234 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.722238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.722279 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.722349 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.722992 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.723746 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.724892 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.725234 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.725341 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.726758 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.726764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.726767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.726821 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.729436 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.729865 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.729869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.729870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.729957 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.730234 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.730238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.730240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.730483 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.730614 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.730895 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.731063 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":62}
I20260810 06:25:33.732161 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.721312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.4.73.62:39809)
0810 06:25:33.721442 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.721446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.722016 (+ 570us) server_negotiation.cc:407] Connection header received
0810 06:25:33.722183 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.722186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.722234 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.722309 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.722954 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.723747 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.725854 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.726189 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.726295 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.728083 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.728087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.728090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.728117 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.730228 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.730774 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.730777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.730779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.730866 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.731188 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.731192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.731194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.731414 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.731476 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.731904 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.732018 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:25:33.732589 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.721223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:50884)
0810 06:25:33.721908 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.721922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.722060 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.722442 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.722445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.722698 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.722827 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.722833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.724407 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.724413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.725661 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.725679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.726320 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.726324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.726425 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.726945 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.727027 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.727922 (+ 895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.730335 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.730337 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.730340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.730677 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.730972 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.730975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.730977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.731075 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.731576 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.731581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.731684 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.731823 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.732459 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":17,"thread_start_us":110,"threads_started":1}
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.62@KRBTEST.COM for oryx/127.4.73.61@KRBTEST.COM
I20260810 06:25:33.745922 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.736616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.0.0.1:55234)
0810 06:25:33.736987 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.737042 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.737173 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.737457 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.737463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.737899 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.738023 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.738030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.738680 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.738683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.739622 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.739650 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.740057 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.740061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.740168 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.740665 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.740680 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.742385 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.744074 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.744084 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.744090 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.744373 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.744720 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.744724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.744726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.744825 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.745287 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.745293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.745414 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.745557 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.745730 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
I20260810 06:25:33.745934 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.736654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.4.73.61:33575)
0810 06:25:33.736763 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.736766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.737134 (+ 368us) server_negotiation.cc:407] Connection header received
0810 06:25:33.737292 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.737295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.737336 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.737402 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.738134 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.738579 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.739770 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.739956 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.740025 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.742537 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.742540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.742542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.742563 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.743950 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.744479 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.744484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.744486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.744582 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.744936 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.744941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.744944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.745170 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.745273 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.745634 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.745753 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:25:33.747455 14705 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.73.61:33575: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26"
member_type: NON_VOTER
last_known_addr {
host: "127.4.73.61"
port: 33575
}
attrs {
promote: true
}
}
dest_uuid: "45105579b0a74e16b191d3ad16b5a7b7"
cas_config_opid_index: -1
I20260810 06:25:33.748162 14705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } }
I20260810 06:25:33.750270 15631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 50a46bcc18584f2a8b0ba2aa73c5ea26 (127.4.73.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } }
I20260810 06:25:33.751044 15545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:33.752137 15631 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.73.61:33575
I20260810 06:25:33.752362 15545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Refusing update from remote peer 45105579b0a74e16b191d3ad16b5a7b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 06:25:33.752638 15632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } }
I20260810 06:25:33.752729 15467 master.cc:537] Master@127.4.73.61:33575 shutting down...
I20260810 06:25:33.752745 15632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:33.753378 15632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.4.73.60@KRBTEST.COM; defaulting to no policy
W20260810 06:25:33.758270 14684 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 06:25:33.759661 14684 consensus_peers.cc:487] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 -> Peer 50a46bcc18584f2a8b0ba2aa73c5ea26 (127.4.73.61:33575): Couldn't send request to peer 50a46bcc18584f2a8b0ba2aa73c5ea26. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.60@KRBTEST.COM" created.
I20260810 06:25:33.760025 15467 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:33.760340 15467 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:33.760462 15467 tablet_replica.cc:330] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: stopping tablet replica
I20260810 06:25:33.766630 15467 master.cc:552] Master@127.4.73.61:33575 shutdown complete.
I20260810 06:25:33.766731 15467 master_runner.cc:305] Clearing existing system tablet
I20260810 06:25:33.767104 15467 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:33.767241 15467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:33.773000 15467 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260810 06:25:33.774632 15644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.774971 15467 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:33.775161 15467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 06:25:33.775209 15467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "69db13f424eec6ed2b86947ef79c0ce2"
server_key_iv: "825314aa94b637f99dd0834ec4be8b1b"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.777042 15467 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.60.keytab.
Entry for principal oryx/127.4.73.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.60.keytab.
I20260810 06:25:33.778733 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:39149
--webserver_interface=127.4.73.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.60.keytab
--principal=oryx/127.4.73.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:25:33.779588 15467 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 06:25:33.779697 15467 log.cc:1196] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 06:25:33.780033 15467 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Deleting consensus metadata
I20260810 06:25:33.780233 15467 master_runner.cc:315] Copying system tablet from 127.4.73.62:39809
I20260810 06:25:33.781886 15467 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: tablet copy: Beginning tablet copy session from remote peer at address 127.4.73.62:39809
I20260810 06:25:33.792544 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.782674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.4.73.62:39809)
0810 06:25:33.782790 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.782795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.783087 (+ 292us) server_negotiation.cc:407] Connection header received
0810 06:25:33.783211 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.783216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.783275 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.783388 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.784079 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.784902 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.786326 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.786733 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.786852 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.788401 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.788405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.788408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.788432 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.790203 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.790786 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.790789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.790791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.790888 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.791244 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.791248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.791249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.791472 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.791569 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.792231 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.792383 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:25:33.792672 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.782611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:50890)
0810 06:25:33.782971 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.783000 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.783122 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.783437 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.783443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.783705 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.783877 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.783888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.785037 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.785042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.786101 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.786116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.786885 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.786892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.787039 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.787642 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.787664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.788171 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.790331 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.790336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.790341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.790666 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.790992 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.790997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.791000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.791134 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.791587 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.791591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.791736 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.792162 (+ 426us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.792401 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260810 06:25:33.793361 14736 tablet_copy_service.cc:140] P 45105579b0a74e16b191d3ad16b5a7b7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 50a46bcc18584f2a8b0ba2aa73c5ea26 ({username='oryx', principal='oryx/127.4.73.61@KRBTEST.COM'} at 127.0.0.1:50890)
I20260810 06:25:33.793553 14736 tablet_copy_service.cc:161] P 45105579b0a74e16b191d3ad16b5a7b7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 50a46bcc18584f2a8b0ba2aa73c5ea26 at {username='oryx', principal='oryx/127.4.73.61@KRBTEST.COM'} at 127.0.0.1:50890: session id = 50a46bcc18584f2a8b0ba2aa73c5ea26-00000000000000000000000000000000
I20260810 06:25:33.795531 14736 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 06:25:33.797523 15467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 06:25:33.800027 15467 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: tablet copy: Starting download of 0 data blocks...
I20260810 06:25:33.800237 15467 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: tablet copy: Starting download of 1 WAL segments...
I20260810 06:25:33.802076 15467 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 06:25:33.803845 14736 tablet_copy_service.cc:342] P 45105579b0a74e16b191d3ad16b5a7b7: Request end of tablet copy session 50a46bcc18584f2a8b0ba2aa73c5ea26-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.73.61@KRBTEST.COM'} at 127.0.0.1:50890
I20260810 06:25:33.803951 14736 tablet_copy_service.cc:433] P 45105579b0a74e16b191d3ad16b5a7b7: ending tablet copy session 50a46bcc18584f2a8b0ba2aa73c5ea26-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 50a46bcc18584f2a8b0ba2aa73c5ea26
I20260810 06:25:33.811619 15467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:33.813360 15664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:33.813567 15660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.813865 15467 server_base.cc:1034] running on GCE node
W20260810 06:25:33.814013 15661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.814280 15467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:33.814576 15467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:33.815732 15467 hybrid_clock.cc:648] HybridClock initialized: now 1786343133815718 us; error 28 us; skew 500 ppm
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:33.817477 15467 init.cc:377] Logged in from keytab as oryx/127.4.73.61@KRBTEST.COM (short username oryx)
I20260810 06:25:33.817919 15467 webserver.cc:460] Webserver started at http://127.4.73.61:36217/ using document root <none> and password file <none>
I20260810 06:25:33.818094 15467 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:33.818126 15467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:33.819499 15467 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 06:25:33.820518 15675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.820742 15467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:33.820850 15467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "69db13f424eec6ed2b86947ef79c0ce2"
server_key_iv: "825314aa94b637f99dd0834ec4be8b1b"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.821003 15467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:33.840890 15467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:33.843140 15467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:33.849984 15467 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.61:33575
I20260810 06:25:33.850028 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.61:33575 every 8 connection(s)
I20260810 06:25:33.850762 15467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
I20260810 06:25:33.852807 15756 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:25:33.853837 15756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Bootstrap starting.
I20260810 06:25:33.864280 15756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:25:33.864631 15756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Bootstrap complete.
I20260810 06:25:33.865206 15756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } }
I20260810 06:25:33.865341 15756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50a46bcc18584f2a8b0ba2aa73c5ea26, State: Initialized, Role: LEARNER
I20260810 06:25:33.865440 15756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } }
I20260810 06:25:33.865778 15759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } }
I20260810 06:25:33.865944 15756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:33.865968 15759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: LEARNER
I20260810 06:25:33.869539 15777 catalog_manager.cc:1259] Loaded cluster ID: a798e26e8a0e4b7ea5f122b7386bd3fc
I20260810 06:25:33.869606 15777 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: loading cluster ID for follower catalog manager: success
I20260810 06:25:33.871722 15777 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: acquiring CA information for follower catalog manager: success
I20260810 06:25:33.873164 15777 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 06:25:33.909591 15645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:33.909860 15645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:33.909927 15645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:33.913815 15645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:33.913888 15645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:33.913920 15645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:33.913977 15645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:33.914011 15645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:33.919473 15645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.60.keytab
--principal=oryx/127.4.73.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:39149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.73.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:33.920559 15645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:33.921710 15645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:33.930487 15783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:33.930487 15788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:33.930629 15786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:33.930905 15645 server_base.cc:1034] running on GCE node
I20260810 06:25:33.931393 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:33.932435 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:33.933594 15645 hybrid_clock.cc:648] HybridClock initialized: now 1786343133933566 us; error 40 us; skew 500 ppm
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:33.936252 15645 init.cc:377] Logged in from keytab as oryx/127.4.73.60@KRBTEST.COM (short username oryx)
I20260810 06:25:33.937140 15645 webserver.cc:460] Webserver started at http://127.4.73.60:33137/ using document root <none> and password file <none>
I20260810 06:25:33.937678 15645 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:33.937740 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:33.938012 15645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:33.940060 15645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/instance:
uuid: "829323b021fd4796b4dc2069177ffdc3"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "15ee6c2c5e2feaf71babf1192626641d"
server_key_iv: "3037446dc3bd653ece2b2168142c6bf1"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.940673 15645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal/instance:
uuid: "829323b021fd4796b4dc2069177ffdc3"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "15ee6c2c5e2feaf71babf1192626641d"
server_key_iv: "3037446dc3bd653ece2b2168142c6bf1"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.944239 15645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 06:25:33.946497 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:33.947719 15645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:25:33.947881 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
uuid: "829323b021fd4796b4dc2069177ffdc3"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "15ee6c2c5e2feaf71babf1192626641d"
server_key_iv: "3037446dc3bd653ece2b2168142c6bf1"
server_key_version: "encryptionkey@0"
I20260810 06:25:33.947992 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:33.964041 15645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:33.967151 15645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:33.967337 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:33.974931 15645 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.60:39149
I20260810 06:25:33.975214 15893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.60:39149 every 8 connection(s)
I20260810 06:25:33.975881 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
I20260810 06:25:33.978673 15894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:33.979270 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 15645
I20260810 06:25:33.983271 15894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.73.60@KRBTEST.COM
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.60@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:33.993625 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.980718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.4.73.60:39149)
0810 06:25:33.981171 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.981186 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.981219 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:25:33.981985 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.982003 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.982318 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.982654 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.983461 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.984633 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.985890 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.986339 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.986516 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.989045 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.989053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.989069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.989094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.990842 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.991296 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.991302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.991306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.991451 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.991747 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.991750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.991752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.992111 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.992373 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.992639 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.992928 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":149,"threads_started":1}
I20260810 06:25:33.995286 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.984896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:50894)
0810 06:25:33.985333 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.985383 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.985505 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.985818 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.985827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.986180 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.986311 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.986317 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.987278 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.987281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.987909 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.987918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.988453 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.988457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.988569 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.989139 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.989152 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:33.990844 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:33.992858 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.992870 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.992875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.993122 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.993424 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:33.993428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:33.993430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:33.993523 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:33.994814 (+ 1291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:33.994819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:33.994915 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 06:25:33.995028 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 06:25:33.995154 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":183,"threads_started":1}
I20260810 06:25:33.995381 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.985131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.4.73.62:39809)
0810 06:25:33.985255 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.985260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.985475 (+ 215us) server_negotiation.cc:407] Connection header received
0810 06:25:33.985607 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.985611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.985659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.985736 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.986441 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.987132 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.988048 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.988273 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.988417 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.990982 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:33.990986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:33.990989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:33.991019 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:33.992719 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.993233 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.993237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.993239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.993328 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:33.993628 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:33.993631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:33.993633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:33.994688 (+ 1055us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:33.994792 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:33.995115 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 06:25:33.995235 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:25:33.996438 15894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:25:33 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.60@KRBTEST.COM for oryx/127.4.73.61@KRBTEST.COM
I20260810 06:25:34.004168 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.996882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.0.0.1:55244)
0810 06:25:33.997023 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:33.997036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:33.997103 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:33.997460 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:33.997463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:33.997611 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:33.997711 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.997717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.998588 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.998590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.999093 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:33.999101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.999526 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.999528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:33.999599 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:33.999970 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:33.999981 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.001146 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.002905 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.002907 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.002908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.003082 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.003324 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.003326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.003327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.003405 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.003731 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.003733 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.003802 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.003882 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.003976 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 06:25:34.004192 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:33.996997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.4.73.61:33575)
0810 06:25:33.997246 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:33.997257 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:33.997270 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:33.997307 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:33.997309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:33.997346 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:33.997422 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:33.997824 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.998490 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.999209 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:33.999432 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:33.999508 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.001278 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.001281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.001283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.001306 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.002812 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.003164 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.003167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.003168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.003247 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.003481 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.003483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.003485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.003651 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.003727 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.003941 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.004069 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
I20260810 06:25:34.005069 15894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343133, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.60@KRBTEST.COM for oryx/127.4.73.60@KRBTEST.COM
I20260810 06:25:34.013119 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.005556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.0.0.1:52176)
0810 06:25:34.005683 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.005698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.005809 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.006036 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.006039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.006188 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.006246 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.006251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.007329 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.007335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.007827 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.007834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.008286 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.008289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.008423 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.008816 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.008826 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.009985 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.011572 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.011577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.011579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.011773 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.012055 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.012060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.012062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.012141 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.012599 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.012602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.012689 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.012798 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.012940 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 06:25:34.013125 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.005601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.4.73.60:39149)
0810 06:25:34.005695 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.005700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.005765 (+ 65us) server_negotiation.cc:407] Connection header received
0810 06:25:34.005903 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.005906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.005947 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.006024 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.006354 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.007215 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.007945 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.008173 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.008286 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.010113 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.010117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.010119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.010149 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.011424 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.011878 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.011883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.011885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.011963 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.012236 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.012239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.012241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.012519 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.012609 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.012848 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.013004 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":44}
I20260810 06:25:34.016530 15894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Bootstrap starting.
I20260810 06:25:34.018893 15894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:34.019649 15894 log.cc:824] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:34.021806 15894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: No bootstrap required, opened a new log
I20260810 06:25:34.024992 15894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } }
I20260810 06:25:34.025224 15894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:34.025321 15894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829323b021fd4796b4dc2069177ffdc3, State: Initialized, Role: FOLLOWER
I20260810 06:25:34.025830 15894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } }
I20260810 06:25:34.026304 15904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } } }
I20260810 06:25:34.026417 15904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:25:34.027009 15894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 06:25:34.032567 15921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:25:34.032652 15921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:25:34.036427 15645 master_runner.cc:428] Detected that this master 829323b021fd4796b4dc2069177ffdc3 is joining an existing cluster
I20260810 06:25:34.036510 15645 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.73.60:39149
I20260810 06:25:34.047013 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 14805
I20260810 06:25:34.058470 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.039524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.4.73.61:33575)
0810 06:25:34.039837 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.039843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.039859 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:25:34.039902 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.039904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.039945 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.040042 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.040667 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.041545 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.042900 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.043259 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.043335 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.045037 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.045040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.045043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.045071 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.046812 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.057158 (+ 10346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.057163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.057165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.057267 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.057571 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.057575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.057576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.057802 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.057918 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.058185 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.058327 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260810 06:25:34.058501 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.039424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.0.0.1:55252)
0810 06:25:34.039710 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.039724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.039821 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.040058 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.040062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.040354 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.040525 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.040536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.041684 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.041687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.042751 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.042761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.043501 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.043504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.043749 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.044445 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.044463 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.044889 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.056783 (+ 11894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.056785 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.056787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.057032 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.057370 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.057373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.057375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.057468 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.057906 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.057909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.058013 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.058115 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.058300 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":123,"spinlock_wait_cycles":4992,"thread_start_us":111,"threads_started":1}
I20260810 06:25:34.061647 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:44507
--local_ip_for_outbound_sockets=127.4.73.1
--tserver_master_addrs=127.4.73.62:39809
--webserver_port=44267
--webserver_interface=127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab
--principal=oryx/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:25:34.065790 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.039831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.4.73.62:39809)
0810 06:25:34.040229 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.040235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.040260 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:25:34.040324 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.040329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.040381 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.040461 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.041057 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.041809 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.042923 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.043265 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.043335 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.061920 (+ 18585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.061924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.061927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.061955 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.063825 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.064435 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.064440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.064445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.064533 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.064846 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.064849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.064851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.065091 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.065181 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.065517 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.065637 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260810 06:25:34.065807 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.038954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:50908)
0810 06:25:34.039249 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.039267 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.039362 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.040584 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.040590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.040802 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.040915 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.040923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.041944 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.041947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.042795 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.042806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.043502 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.043505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.043738 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.061091 (+ 17353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.061106 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.061733 (+ 627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.063966 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.063972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.063974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.064322 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.064650 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.064653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.064655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.064743 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.065224 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.065228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.065343 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.065458 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.065606 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":77,"spinlock_wait_cycles":768,"thread_start_us":124,"threads_started":1}
I20260810 06:25:34.066810 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.040766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.0.0.1:52192)
0810 06:25:34.041053 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.041066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.041182 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.041470 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.041473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.041682 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.041848 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.041856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.043001 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.043004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.043869 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.043879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.044503 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.044506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.044601 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.061052 (+ 16451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.061070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.061733 (+ 663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.064406 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.064410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.064413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.064664 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.064967 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.064970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.064971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.065054 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.066273 (+ 1219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.066277 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.066367 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.066483 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.066617 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":6,"thread_start_us":86,"threads_started":1}
I20260810 06:25:34.066825 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.040599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.4.73.60:39149)
0810 06:25:34.040700 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.040705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.041131 (+ 426us) server_negotiation.cc:407] Connection header received
0810 06:25:34.041269 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.041272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.041305 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.041351 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.041964 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.042869 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.043977 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.044371 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.044476 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.061903 (+ 17427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.061907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.061909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.061932 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.064260 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.064757 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.064775 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.064780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.064864 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.065149 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.065154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.065156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.065725 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.065841 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.066574 (+ 733us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.066695 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":340,"server-negotiator.queue_time_us":42}
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343132, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.62@KRBTEST.COM for oryx/127.4.73.60@KRBTEST.COM
I20260810 06:25:34.077819 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.069912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.0.0.1:52206)
0810 06:25:34.070077 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.070091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.070206 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.070518 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.070521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.070707 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.070791 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.070797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.071510 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.071513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.072057 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.072065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.072674 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.072678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.072783 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.073228 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.073248 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.074514 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.076238 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.076243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.076245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.076503 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.076809 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.076812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.076814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.076904 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.077272 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.077275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.077378 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.077502 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.077647 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 06:25:34.077819 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.069952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.4.73.60:39149)
0810 06:25:34.070060 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.070064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.070167 (+ 103us) server_negotiation.cc:407] Connection header received
0810 06:25:34.070344 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.070347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.070400 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.070502 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.070910 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.071403 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.072173 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.072562 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.072671 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.074649 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.074652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.074655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.074696 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.076120 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.076607 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.076611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.076613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.076692 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.077005 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.077008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.077010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.077172 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.077287 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.077566 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.077672 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 06:25:34.079105 14705 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.73.60:39149: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "829323b021fd4796b4dc2069177ffdc3"
member_type: NON_VOTER
last_known_addr {
host: "127.4.73.60"
port: 39149
}
attrs {
promote: true
}
}
dest_uuid: "45105579b0a74e16b191d3ad16b5a7b7"
cas_config_opid_index: 5
I20260810 06:25:34.079779 14705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } }
I20260810 06:25:34.080936 15632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.082261 15632 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.73.60:39149
I20260810 06:25:34.082432 15632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.082558 15632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:34.083098 15645 master.cc:537] Master@127.4.73.60:39149 shutting down...
I20260810 06:25:34.083415 15855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:34.085192 15855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Refusing update from remote peer 45105579b0a74e16b191d3ad16b5a7b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 06:25:34.086179 15631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 06:25:34.087936 14688 consensus_peers.cc:487] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 -> Peer 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60:39149): Couldn't send request to peer 829323b021fd4796b4dc2069177ffdc3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 06:25:34.091078 15645 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:34.091367 15645 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:34.091488 15645 tablet_replica.cc:330] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: stopping tablet replica
I20260810 06:25:34.095314 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.081342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.4.73.61:33575)
0810 06:25:34.085011 (+ 3669us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.085017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.089961 (+ 4944us) server_negotiation.cc:407] Connection header received
0810 06:25:34.090021 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.090024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.090089 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.090190 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:34.090497 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.091329 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.093072 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.094601 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.094727 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.094843 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.095122 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.095194 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3564}
I20260810 06:25:34.095314 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.081063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.0.0.1:55266)
0810 06:25:34.089743 (+ 8680us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.089771 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.089870 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.090213 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.090216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.090233 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:34.090353 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.090361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.091599 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.091603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.092928 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.092940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.094898 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.094900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.094967 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.095000 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.095104 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.095178 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8598}
I20260810 06:25:34.096195 15717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 LEARNER]: Refusing update from remote peer 45105579b0a74e16b191d3ad16b5a7b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 06:25:34.096824 15631 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 06:25:34.099112 15645 master.cc:552] Master@127.4.73.60:39149 shutdown complete.
I20260810 06:25:34.099206 15645 master_runner.cc:305] Clearing existing system tablet
I20260810 06:25:34.098851 15717 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.099658 15645 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:34.099757 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:34.101104 15645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:34.101418 15759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45105579b0a74e16b191d3ad16b5a7b7. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.101523 15759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: LEARNER
I20260810 06:25:34.101608 15759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: NON_VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: true } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.101689 15759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: LEARNER
I20260810 06:25:34.102138 15939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.102387 15645 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:34.102509 15645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:34.102546 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
uuid: "829323b021fd4796b4dc2069177ffdc3"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "15ee6c2c5e2feaf71babf1192626641d"
server_key_iv: "3037446dc3bd653ece2b2168142c6bf1"
server_key_version: "encryptionkey@0"
I20260810 06:25:34.103060 15631 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: attempting to promote NON_VOTER 50a46bcc18584f2a8b0ba2aa73c5ea26 to VOTER
I20260810 06:25:34.103487 15631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } }
I20260810 06:25:34.104655 15645 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:34.105367 15717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 LEARNER]: Refusing update from remote peer 45105579b0a74e16b191d3ad16b5a7b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 06:25:34.105731 15632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 06:25:34.107093 15645 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 06:25:34.107179 15645 log.cc:1196] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 06:25:34.107465 15645 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Deleting consensus metadata
I20260810 06:25:34.107649 15645 master_runner.cc:315] Copying system tablet from 127.4.73.62:39809
I20260810 06:25:34.107527 15632 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50a46bcc18584f2a8b0ba2aa73c5ea26 (127.4.73.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.108706 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.104645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.0.0.1:52214)
0810 06:25:34.108542 (+ 3897us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.108626 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3821}
I20260810 06:25:34.108736 15717 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 50a46bcc18584f2a8b0ba2aa73c5ea26 (127.4.73.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.109843 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.109457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.0.0.1:52226)
0810 06:25:34.109719 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.109770 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
W20260810 06:25:34.110045 14688 consensus_peers.cc:487] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 -> Peer 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60:39149): Couldn't send request to peer 829323b021fd4796b4dc2069177ffdc3. Status: Network error: Client connection negotiation failed: client connection to 127.4.73.60:39149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 06:25:34.110219 15759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.110342 15759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:25:34.110827 15645 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: tablet copy: Beginning tablet copy session from remote peer at address 127.4.73.62:39809
I20260810 06:25:34.111605 15632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.111747 15632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:34.111891 15632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.111999 15632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:34.119944 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.111883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:50910)
0810 06:25:34.112686 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.112696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.112832 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.113089 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.113092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.113290 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.113401 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.113407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.114164 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.114169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.114855 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.114864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.115354 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.115358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.115443 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.116060 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.116103 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.116595 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.118378 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.118383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.118386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.118600 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.118894 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.118897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.118899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.118990 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.119372 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.119375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.119481 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.119592 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.119740 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":59,"threads_started":1}
I20260810 06:25:34.119989 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.111575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.4.73.62:39809)
0810 06:25:34.111672 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.111676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.112785 (+ 1109us) server_negotiation.cc:407] Connection header received
0810 06:25:34.112922 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.112925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.112970 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.113056 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.113519 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.114035 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.114979 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.115229 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.115349 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.116728 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.116732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.116735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.116762 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.118253 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.118707 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.118711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.118713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.118800 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.119083 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.119087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.119089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.119270 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.119382 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.119728 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.119852 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:25:34.120632 14736 tablet_copy_service.cc:140] P 45105579b0a74e16b191d3ad16b5a7b7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 829323b021fd4796b4dc2069177ffdc3 ({username='oryx', principal='oryx/127.4.73.60@KRBTEST.COM'} at 127.0.0.1:50910)
I20260810 06:25:34.120771 14736 tablet_copy_service.cc:161] P 45105579b0a74e16b191d3ad16b5a7b7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 829323b021fd4796b4dc2069177ffdc3 at {username='oryx', principal='oryx/127.4.73.60@KRBTEST.COM'} at 127.0.0.1:50910: session id = 829323b021fd4796b4dc2069177ffdc3-00000000000000000000000000000000
I20260810 06:25:34.121547 14736 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 06:25:34.122725 15645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 06:25:34.124687 15645 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: tablet copy: Starting download of 0 data blocks...
I20260810 06:25:34.124902 15645 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: tablet copy: Starting download of 1 WAL segments...
I20260810 06:25:34.126276 15645 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 06:25:34.127910 14736 tablet_copy_service.cc:342] P 45105579b0a74e16b191d3ad16b5a7b7: Request end of tablet copy session 829323b021fd4796b4dc2069177ffdc3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.73.60@KRBTEST.COM'} at 127.0.0.1:50910
I20260810 06:25:34.127991 14736 tablet_copy_service.cc:433] P 45105579b0a74e16b191d3ad16b5a7b7: ending tablet copy session 829323b021fd4796b4dc2069177ffdc3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 829323b021fd4796b4dc2069177ffdc3
I20260810 06:25:34.134900 15645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:34.136538 15952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:34.141275 15953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:34.141609 15645 server_base.cc:1034] running on GCE node
W20260810 06:25:34.141765 15955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:34.142032 15645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:34.142403 15645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:34.143548 15645 hybrid_clock.cc:648] HybridClock initialized: now 1786343134143556 us; error 43 us; skew 500 ppm
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:34.145141 15645 init.cc:377] Logged in from keytab as oryx/127.4.73.60@KRBTEST.COM (short username oryx)
I20260810 06:25:34.145632 15645 webserver.cc:460] Webserver started at http://127.4.73.60:46027/ using document root <none> and password file <none>
I20260810 06:25:34.145849 15645 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:34.145907 15645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:34.147046 15645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 06:25:34.148008 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.148249 15645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 06:25:34.148371 15645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
uuid: "829323b021fd4796b4dc2069177ffdc3"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "15ee6c2c5e2feaf71babf1192626641d"
server_key_iv: "3037446dc3bd653ece2b2168142c6bf1"
server_key_version: "encryptionkey@0"
I20260810 06:25:34.148463 15645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:34.156803 15645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:34.158828 15645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:34.163820 15645 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.60:39149
I20260810 06:25:34.163829 16050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.60:39149 every 8 connection(s)
I20260810 06:25:34.164438 15645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
I20260810 06:25:34.165881 16051 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:25:34.166954 16051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Bootstrap starting.
I20260810 06:25:34.175519 16051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:25:34.175829 16051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: Bootstrap complete.
I20260810 06:25:34.176367 16051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } }
I20260810 06:25:34.176506 16051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 829323b021fd4796b4dc2069177ffdc3, State: Initialized, Role: LEARNER
I20260810 06:25:34.176623 16051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } }
I20260810 06:25:34.176970 16058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.177134 16051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:34.177174 16058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: This master's current role is: LEARNER
I20260810 06:25:34.181424 16074 catalog_manager.cc:1259] Loaded cluster ID: a798e26e8a0e4b7ea5f122b7386bd3fc
I20260810 06:25:34.181510 16074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: loading cluster ID for follower catalog manager: success
I20260810 06:25:34.183032 16074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: acquiring CA information for follower catalog manager: success
I20260810 06:25:34.184134 16074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 06:25:34.223940 15935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:34.224264 15935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:34.224342 15935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:34.230643 15935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:34.230716 15935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:34.230813 15935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:34.239565 15935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab
--principal=oryx/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:44507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=44267
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:34.240754 15935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:34.241961 15935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:25:34.253520 15935 server_base.cc:1034] running on GCE node
W20260810 06:25:34.253448 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:34.253448 16087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:34.253556 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:34.254117 15935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:34.254695 15935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:34.255875 15935 hybrid_clock.cc:648] HybridClock initialized: now 1786343134255862 us; error 26 us; skew 500 ppm
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:34.258909 15935 init.cc:377] Logged in from keytab as oryx/127.4.73.1@KRBTEST.COM (short username oryx)
I20260810 06:25:34.259775 15935 webserver.cc:460] Webserver started at http://127.4.73.1:44267/ using document root <none> and password file <none>
I20260810 06:25:34.259923 15219 heartbeater.cc:503] Master 127.4.73.62:39809 was elected leader, sending a full tablet report...
I20260810 06:25:34.260408 15935 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:34.260517 15935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:34.265565 15935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:25:34.268539 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.270213 15935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:25:34.270350 15935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "750304f89b244671bb6de42e1cfa4c84"
format_stamp: "Formatted at 2026-08-10 06:25:32 on dist-test-slave-45dx"
server_key: "2ce944e0bfbc054d1777660a7bc695ae"
server_key_iv: "f8b02f9004dfd0d0530ce206023cc9be"
server_key_version: "encryptionkey@0"
I20260810 06:25:34.271162 15935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:34.296919 15935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:34.300506 15935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:34.300714 15935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:34.301344 15935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:34.302464 15935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:34.302621 15935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.302745 15935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:34.302783 15935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.317750 15935 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:44507
I20260810 06:25:34.317832 16274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:44507 every 8 connection(s)
I20260810 06:25:34.318871 15935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:34.323189 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 15935
I20260810 06:25:34.323405 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 15011
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.61@KRBTEST.COM
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.60@KRBTEST.COM
I20260810 06:25:34.340788 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.321037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:56387 (local address 127.4.73.61:33575)
0810 06:25:34.321191 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.321196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.322538 (+ 1342us) server_negotiation.cc:407] Connection header received
0810 06:25:34.326330 (+ 3792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.326334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.326387 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.326472 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.327312 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.328094 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.330004 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.330356 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.330435 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.334398 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.334403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.334407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.334439 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.336253 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.338712 (+ 2459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.338715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.338717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.338849 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.339301 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.339304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.339307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.339564 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.339653 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.340459 (+ 806us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.340591 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:25:34.342859 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.321644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.4.73.1:56387)
0810 06:25:34.322315 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.322366 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.326185 (+ 3819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.326697 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.326700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.326984 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.327171 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.327181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.328425 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.328428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.329870 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.329881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.330639 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.330642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.330848 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.331567 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.331587 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.334188 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.336438 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.336444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.336461 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.336930 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.338990 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.338993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.338998 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.339191 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.339736 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.339756 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.340128 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.341178 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.341578 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":50,"spinlock_wait_cycles":57472,"thread_start_us":161,"threads_started":1}
I20260810 06:25:34.344337 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.322935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:52103 (local address 127.4.73.62:39809)
0810 06:25:34.323089 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.323094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.323392 (+ 298us) server_negotiation.cc:407] Connection header received
0810 06:25:34.323921 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.323927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.323985 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.324037 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.328676 (+ 4639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.329492 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.333278 (+ 3786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.333649 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.333737 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.337193 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.337197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.337200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.337233 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.339105 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.342872 (+ 3767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.342879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.342883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.342993 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.343357 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.343361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.343362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.343592 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.343678 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.344011 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.344131 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 06:25:34.345995 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.321217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:40087 (local address 127.4.73.60:39149)
0810 06:25:34.321559 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.321563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.322473 (+ 910us) server_negotiation.cc:407] Connection header received
0810 06:25:34.325661 (+ 3188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.325666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.325728 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.325789 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.326622 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.327818 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.331872 (+ 4054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.332324 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.332413 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.339257 (+ 6844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.339264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.339270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.339303 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.341090 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.343529 (+ 2439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.343532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.343534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.343623 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.343945 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.343949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.343951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.344177 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.344260 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.345721 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.345852 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":165,"threads_started":1}
I20260810 06:25:34.344964 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.322782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:52103)
0810 06:25:34.323289 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.323306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.323814 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.324436 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.324445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.324892 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.325486 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.325503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.330550 (+ 5047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.330555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.331692 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.331707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.333920 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.333924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.334012 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.335120 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.335136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.337019 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.342476 (+ 5457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.342479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.342480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.342721 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.343125 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.343128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.343129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.343219 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.343732 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.343735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.343826 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.344546 (+ 720us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.344696 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":195,"threads_started":1}
I20260810 06:25:34.346006 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.321644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.4.73.1:40087)
0810 06:25:34.322315 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.322369 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.323643 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.325955 (+ 2312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.325960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.326298 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.326479 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.326487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.328094 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.328098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.330412 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.330422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.332603 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.332606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.332851 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.334547 (+ 1696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.334563 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.338554 (+ 3991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.343180 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.343183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.343185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.343417 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.343732 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.343735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.343737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.343829 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.344340 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.344345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.345520 (+ 1175us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.345646 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.345793 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1062,"spinlock_wait_cycles":768,"thread_start_us":183,"threads_started":1}
I20260810 06:25:34.348083 16278 heartbeater.cc:347] Connected to a master server at 127.4.73.61:33575
I20260810 06:25:34.348083 16277 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39809
I20260810 06:25:34.348390 16278 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.348390 16277 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.348551 16276 heartbeater.cc:347] Connected to a master server at 127.4.73.60:39149
I20260810 06:25:34.348675 16276 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.349045 16278 heartbeater.cc:511] Master 127.4.73.61:33575 requested a full tablet report, sending...
I20260810 06:25:34.350580 15700 ts_manager.cc:194] Registered new tserver with Master: 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507)
I20260810 06:25:34.349046 16276 heartbeater.cc:511] Master 127.4.73.60:39149 requested a full tablet report, sending...
I20260810 06:25:34.353734 15992 ts_manager.cc:194] Registered new tserver with Master: 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507)
I20260810 06:25:34.354133 16277 heartbeater.cc:511] Master 127.4.73.62:39809 requested a full tablet report, sending...
I20260810 06:25:34.354991 14705 ts_manager.cc:194] Re-registered known tserver with Master: 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507)
I20260810 06:25:34.355835 14705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:52103
I20260810 06:25:34.360663 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:35799
--local_ip_for_outbound_sockets=127.4.73.2
--tserver_master_addrs=127.4.73.62:39809
--webserver_port=42149
--webserver_interface=127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab
--principal=oryx/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:25:34.469841 15458 heartbeater.cc:503] Master 127.4.73.62:39809 was elected leader, sending a full tablet report...
W20260810 06:25:34.477649 16291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:34.477936 16291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:34.478032 16291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:34.482378 16291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:34.482429 16291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:34.482527 16291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:34.487422 16291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab
--principal=oryx/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:35799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=42149
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:34.488425 16291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:34.489390 16291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:34.499220 16298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:34.499181 16297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:34.499294 16303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:34.499526 16291 server_base.cc:1034] running on GCE node
I20260810 06:25:34.499938 16291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:34.500589 16291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:34.501780 16291 hybrid_clock.cc:648] HybridClock initialized: now 1786343134501770 us; error 30 us; skew 500 ppm
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:34.504611 16291 init.cc:377] Logged in from keytab as oryx/127.4.73.2@KRBTEST.COM (short username oryx)
I20260810 06:25:34.505347 16291 webserver.cc:460] Webserver started at http://127.4.73.2:42149/ using document root <none> and password file <none>
I20260810 06:25:34.505868 16291 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:34.505939 16291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:34.509859 16291 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:34.511778 16316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.512980 16291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:34.513099 16291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "1445e6450e8c4466a7f82873e98327a4"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "8f7565aab5cddc2672c656f9edb26454"
server_key_iv: "9fe2eb3e92936523672601d3efe07eb9"
server_key_version: "encryptionkey@0"
I20260810 06:25:34.513616 16291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:34.540042 16291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:34.543212 16291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:34.543416 16291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:34.543978 16291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:34.544823 16291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:34.544869 16291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.544929 16291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:34.544978 16291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.552026 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.546839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.4.73.60:39149)
0810 06:25:34.547009 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.547016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.547116 (+ 100us) server_negotiation.cc:407] Connection header received
0810 06:25:34.547262 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.547288 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.547372 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.547475 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:34.547927 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.548712 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.550298 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.551043 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.551192 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.551396 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.551780 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.551881 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:25:34.552029 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.546729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.0.0.1:52236)
0810 06:25:34.547015 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.547034 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.547148 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.547615 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.547621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.547645 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:34.547770 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.547780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.548865 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.548870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.550122 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.550138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.551477 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.551484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.551612 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.551646 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.551743 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.551840 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260810 06:25:34.553117 16008 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 06:25:34.553557 16058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45105579b0a74e16b191d3ad16b5a7b7. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.553725 16058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: This master's current role is: LEARNER
I20260810 06:25:34.557772 16291 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:35799
I20260810 06:25:34.557818 16499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:35799 every 8 connection(s)
I20260810 06:25:34.558790 16291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:34.569046 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 16291
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.61@KRBTEST.COM
I20260810 06:25:34.569267 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 15228
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.60@KRBTEST.COM
I20260810 06:25:34.583413 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.560577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:55945 (local address 127.4.73.62:39809)
0810 06:25:34.560681 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.560685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.561941 (+ 1256us) server_negotiation.cc:407] Connection header received
0810 06:25:34.562974 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.562977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.563039 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.563089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.565004 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.565471 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.566620 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.566951 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.567066 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.579160 (+ 12094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.579167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.579172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.579203 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.580993 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.581607 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.581613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.581617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.581697 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.582117 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.582122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.582127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.582352 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.582419 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.583077 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.583201 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:25:34.585071 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.561000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:55945)
0810 06:25:34.561774 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.561856 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.562762 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.563305 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.563308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.563722 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.564241 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.564257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.565603 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.565606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.566502 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.566511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.567071 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.567074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.567181 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.567805 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.567820 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.578967 (+ 11147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.581140 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.581148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.581166 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.581502 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.581824 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.581829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.581834 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.582017 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.582496 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.582504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.582801 (+ 297us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.583717 (+ 916us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.584056 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":170,"spinlock_wait_cycles":21632,"thread_start_us":234,"threads_started":1}
I20260810 06:25:34.586305 16503 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39809
I20260810 06:25:34.586506 16503 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.586997 16503 heartbeater.cc:511] Master 127.4.73.62:39809 requested a full tablet report, sending...
I20260810 06:25:34.588038 14705 ts_manager.cc:194] Re-registered known tserver with Master: 1445e6450e8c4466a7f82873e98327a4 (127.4.73.2:35799)
I20260810 06:25:34.588758 14705 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:55945
I20260810 06:25:34.606949 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.562198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.4.73.2:43969)
0810 06:25:34.562453 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.562466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.562760 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.563300 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.563303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.563724 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.564241 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.564257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.565004 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.565007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.566122 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.566132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.566840 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.566842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.566985 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.567627 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.567643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.602831 (+ 35188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.604990 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.604997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.605002 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.605320 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.605682 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.605688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.605693 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.605820 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.606304 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.606307 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.606427 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.606539 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.606709 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":105,"spinlock_wait_cycles":38656,"thread_start_us":102,"threads_started":1}
I20260810 06:25:34.607021 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.562324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:43969 (local address 127.4.73.61:33575)
0810 06:25:34.562449 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.562454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.562529 (+ 75us) server_negotiation.cc:407] Connection header received
0810 06:25:34.562975 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.562978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.563039 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.563089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.564403 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.564888 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.566259 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.566613 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.566683 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.603050 (+ 36367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.603056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.603061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.603098 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.604866 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.605450 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.605455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.605460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.605562 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.605942 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.605948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.605952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.606214 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.606318 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.606625 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.606768 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:25:34.607658 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.561424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:56361 (local address 127.4.73.60:39149)
0810 06:25:34.561526 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.561530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.561938 (+ 408us) server_negotiation.cc:407] Connection header received
0810 06:25:34.562951 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.562954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.563014 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.563112 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.565604 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.566062 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.567147 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.567532 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.567616 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.603867 (+ 36251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.603874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.603880 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.603917 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.605709 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.606247 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.606254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.606258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.606352 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.606672 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.606678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.606679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.606905 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.606983 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.607367 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.607500 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:25:34.608048 16505 heartbeater.cc:347] Connected to a master server at 127.4.73.61:33575
I20260810 06:25:34.608150 16505 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.608196 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.561000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.4.73.2:56361)
0810 06:25:34.561773 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.561847 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.562744 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.563271 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.563278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.563709 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.564240 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.564252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.566209 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.566212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.567009 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.567021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.567817 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.567820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.567893 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.568386 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.568397 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.603683 (+ 35286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.605829 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.605832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.605835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.606133 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.606457 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.606460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.606462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.606556 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.607063 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.607066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.607163 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.607802 (+ 639us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.607968 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":41,"spinlock_wait_cycles":5376,"thread_start_us":259,"threads_started":1}
I20260810 06:25:34.608435 16505 heartbeater.cc:511] Master 127.4.73.61:33575 requested a full tablet report, sending...
I20260810 06:25:34.609124 16502 heartbeater.cc:347] Connected to a master server at 127.4.73.60:39149
I20260810 06:25:34.609231 16502 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.609349 15700 ts_manager.cc:194] Registered new tserver with Master: 1445e6450e8c4466a7f82873e98327a4 (127.4.73.2:35799)
I20260810 06:25:34.609398 16502 heartbeater.cc:511] Master 127.4.73.60:39149 requested a full tablet report, sending...
I20260810 06:25:34.610266 15992 ts_manager.cc:194] Registered new tserver with Master: 1445e6450e8c4466a7f82873e98327a4 (127.4.73.2:35799)
I20260810 06:25:34.610355 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:35931
--local_ip_for_outbound_sockets=127.4.73.3
--tserver_master_addrs=127.4.73.62:39809
--webserver_port=43045
--webserver_interface=127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab
--principal=oryx/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:34.729548 16513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:34.729807 16513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:34.729851 16513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:34.734105 16513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:34.734172 16513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:34.734231 16513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:34.739573 16513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab
--principal=oryx/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:35931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=43045
--tserver_master_addrs=127.4.73.62:39809,127.4.73.61:33575,127.4.73.60:39149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:34.740468 16513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:34.741346 16513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:34.751103 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:34.751093 16519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:34.751098 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:34.751396 16513 server_base.cc:1034] running on GCE node
I20260810 06:25:34.751902 16513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:34.752614 16513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:34.753803 16513 hybrid_clock.cc:648] HybridClock initialized: now 1786343134753780 us; error 30 us; skew 500 ppm
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:34.756639 16513 init.cc:377] Logged in from keytab as oryx/127.4.73.3@KRBTEST.COM (short username oryx)
I20260810 06:25:34.757509 16513 webserver.cc:460] Webserver started at http://127.4.73.3:43045/ using document root <none> and password file <none>
I20260810 06:25:34.758042 16513 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:34.758144 16513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:34.761940 16513 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:25:34.764079 16535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.765288 16513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:34.765410 16513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "86ff81b122d944febf645f52d50d283f"
format_stamp: "Formatted at 2026-08-10 06:25:33 on dist-test-slave-45dx"
server_key: "9f3141dc21fa64a38e30e337be3ac19a"
server_key_iv: "2680eb031b0b011dbd14f24e642f30e8"
server_key_version: "encryptionkey@0"
I20260810 06:25:34.765977 16513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:34.787914 16513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:34.790912 16513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:34.791127 16513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:34.791772 16513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:34.792704 16513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:34.792757 16513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.792832 16513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:34.792862 16513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:34.801970 16513 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:35931
I20260810 06:25:34.802083 16720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:35931 every 8 connection(s)
I20260810 06:25:34.803020 16513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:34.806856 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 16513
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.60@KRBTEST.COM
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
Aug 10 06:25:34 dist-test-slave-45dx krb5kdc[14637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343134, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.61@KRBTEST.COM
I20260810 06:25:34.822407 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.804905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:59581 (local address 127.4.73.60:39149)
0810 06:25:34.805054 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.805059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.806094 (+ 1035us) server_negotiation.cc:407] Connection header received
0810 06:25:34.807308 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.807313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.807393 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.807459 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.809666 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.810541 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.812692 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.813044 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.813120 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.817137 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.817141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.817144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.817183 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.818938 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.819743 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.819747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.819749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.819849 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.820655 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.820661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.820665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.820916 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.821037 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.821879 (+ 842us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.822190 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260810 06:25:34.824182 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.805406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.4.73.3:59581)
0810 06:25:34.805938 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.805980 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.807047 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.807997 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.808006 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.808717 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.809519 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.809542 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.810804 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.810808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.812553 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.812565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.813318 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.813322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.813586 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.814606 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.814628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.816932 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.819116 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.819160 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.819176 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.819549 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.820329 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.820335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.820340 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.820530 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.821112 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.821118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.821442 (+ 324us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.822606 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.822993 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":98,"spinlock_wait_cycles":57216,"thread_start_us":173,"threads_started":1}
I20260810 06:25:34.824376 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.809159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.4.73.60:39149)
0810 06:25:34.811382 (+ 2223us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.811386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.811562 (+ 176us) server_negotiation.cc:407] Connection header received
0810 06:25:34.811682 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.811685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.811735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.811840 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.812750 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.813559 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.815166 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.816199 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.817049 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.818985 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.818990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.818993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.819023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.822106 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.822716 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.822719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.822721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.822809 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.823145 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.823151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.823153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.823368 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.823488 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.824052 (+ 564us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.824180 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2130,"thread_start_us":84,"threads_started":1}
I20260810 06:25:34.825706 16723 heartbeater.cc:347] Connected to a master server at 127.4.73.60:39149
I20260810 06:25:34.825953 16723 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.826545 16723 heartbeater.cc:511] Master 127.4.73.60:39149 requested a full tablet report, sending...
I20260810 06:25:34.826571 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.804911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:45085 (local address 127.4.73.61:33575)
0810 06:25:34.805083 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.805087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.807633 (+ 2546us) server_negotiation.cc:407] Connection header received
0810 06:25:34.807726 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.807730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.807795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.807898 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.809751 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.810594 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.813932 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.814245 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.814348 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.820161 (+ 5813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.820165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.820171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.820200 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.823222 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.825237 (+ 2015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.825240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.825242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.825346 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.825657 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.825660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.825661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.825866 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.825944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.826308 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.826416 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"spinlock_wait_cycles":2432}
I20260810 06:25:34.826576 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.809189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.4.73.62:39809)
0810 06:25:34.810707 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.810722 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.810738 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:25:34.810776 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.810778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.810822 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.810959 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.812087 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.812848 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.814036 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.814631 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.814729 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.817843 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.817848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.817851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.817882 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.823488 (+ 5606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.824029 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.824033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.824034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.824122 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.825307 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.825311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.825312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.825541 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.825663 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.826296 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.826423 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":263,"server-negotiator.queue_time_us":1431,"thread_start_us":95,"threads_started":1}
I20260810 06:25:34.827091 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.806793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.4.73.3:45085)
0810 06:25:34.807519 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.807533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.807632 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.808088 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.808093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.809443 (+ 1350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.809592 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.809601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.810774 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.810777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.812906 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.812917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.814830 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.814834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.815052 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.815809 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.815829 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.820014 (+ 4185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.824889 (+ 4875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.824892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.824893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.825125 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.825468 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.825470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.825472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.825555 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.826006 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.826009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.826119 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.826760 (+ 641us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.826920 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":761,"thread_start_us":105,"threads_started":1}
I20260810 06:25:34.827260 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.809646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.4.73.61:33575)
0810 06:25:34.810248 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.810260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.814425 (+ 4165us) server_negotiation.cc:407] Connection header received
0810 06:25:34.814640 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.814643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.814710 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.814776 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.817283 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.818145 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.819316 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.819733 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.819817 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.822368 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.822372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.822374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.822400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.825337 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.825844 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.825847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.825849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.825938 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.826269 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.826272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.826274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.826473 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.826590 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.826950 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.827079 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":72,"threads_started":1}
I20260810 06:25:34.827806 16726 heartbeater.cc:347] Connected to a master server at 127.4.73.61:33575
I20260810 06:25:34.827903 16726 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.828064 16726 heartbeater.cc:511] Master 127.4.73.61:33575 requested a full tablet report, sending...
I20260810 06:25:34.828862 15700 ts_manager.cc:194] Registered new tserver with Master: 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931)
I20260810 06:25:34.829097 15992 ts_manager.cc:194] Registered new tserver with Master: 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931)
I20260810 06:25:34.829769 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.807333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:40459 (local address 127.4.73.62:39809)
0810 06:25:34.807646 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.807651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.808014 (+ 363us) server_negotiation.cc:407] Connection header received
0810 06:25:34.808186 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.808190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.808247 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.808324 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.809804 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.810595 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.813965 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.814315 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.814408 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.819961 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.819965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.819968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.819994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.821764 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.824906 (+ 3142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.824910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.824912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.825006 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.828848 (+ 3842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.828852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.828853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.829083 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.829162 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.829523 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.829639 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":223}
I20260810 06:25:34.830194 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.807195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:40459)
0810 06:25:34.807933 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.807947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.808072 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.808490 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.808493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.808771 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:34.809688 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.809695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.810949 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.810952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.813778 (+ 2826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.813788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.815452 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.815456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.815570 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.816437 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:34.816451 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:34.819783 (+ 3332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:34.824549 (+ 4766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.824553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.824554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.824781 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.828629 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:34.828631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:34.828633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:34.828734 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:34.829225 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:34.829228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:34.829334 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.829891 (+ 557us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.830035 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":1728,"thread_start_us":91,"threads_started":1}
I20260810 06:25:34.831842 16725 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39809
I20260810 06:25:34.831938 16725 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:34.832108 16725 heartbeater.cc:511] Master 127.4.73.62:39809 requested a full tablet report, sending...
I20260810 06:25:34.832667 14704 ts_manager.cc:194] Re-registered known tserver with Master: 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931)
I20260810 06:25:34.829617 14705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:25:34.833262 14704 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:40459
W20260810 06:25:34.834022 14705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:34.836565 16752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: NON_VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: true } } }
I20260810 06:25:34.836696 16752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:34.839617 16751 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: attempting to promote NON_VOTER 829323b021fd4796b4dc2069177ffdc3 to VOTER
I20260810 06:25:34.839943 16751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:34.840943 16008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 LEARNER]: Refusing update from remote peer 45105579b0a74e16b191d3ad16b5a7b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 06:25:34.841017 15717 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Refusing update from remote peer 45105579b0a74e16b191d3ad16b5a7b7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 06:25:34.841387 16752 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 06:25:34.841507 15949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 06:25:34.843339 15949 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.843622 15717 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.844964 16752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.845146 16752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:34.845016 15949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.845335 15949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:34.845299 16752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.845546 16752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:34.845804 16008 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.846083 16772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.846216 16772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:25:34.849148 16058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45105579b0a74e16b191d3ad16b5a7b7" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:34.849275 16058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:25:34.856552 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.849350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:35799 (local address 127.0.0.1:50474)
0810 06:25:34.849547 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.849577 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.849667 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.851997 (+ 2330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.852000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.852017 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:34.852129 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.852134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.853404 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.853408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.854840 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.854858 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.856119 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.856125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.856228 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.856257 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.856343 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.856418 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":177}
I20260810 06:25:34.857193 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.850009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:35931 (local address 127.0.0.1:53420)
0810 06:25:34.851134 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.851149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.851332 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.852044 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.852046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.852063 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:34.852260 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.852264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.854183 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.854186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.855370 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.855381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.856763 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.856767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.856903 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.856925 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.857002 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.857064 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":1,"thread_start_us":157,"threads_started":1}
I20260810 06:25:34.857538 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.849877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.4.73.1:44507)
0810 06:25:34.850751 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.850760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.850784 (+ 24us) server_negotiation.cc:407] Connection header received
0810 06:25:34.850871 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.850883 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.851040 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.851198 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:34.851453 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.852565 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.855013 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.855904 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.856794 (+ 890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.856945 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.857311 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.857421 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":91,"threads_started":1}
I20260810 06:25:34.857722 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.850450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.4.73.3:35931)
0810 06:25:34.850924 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.850935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.851258 (+ 323us) server_negotiation.cc:407] Connection header received
0810 06:25:34.851473 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.851478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.851654 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.851848 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:34.852374 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.854048 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.855514 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.856436 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.856549 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.856701 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.857412 (+ 711us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.857517 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":167,"threads_started":1}
I20260810 06:25:34.857601 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.849549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.4.73.2:35799)
0810 06:25:34.851486 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.851499 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.851519 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:25:34.851591 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.851595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.851768 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.852137 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:34.852261 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.853280 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.855012 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.855941 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.856926 (+ 985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.857066 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.857116 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.857235 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1820,"thread_start_us":145,"threads_started":1}
I20260810 06:25:34.857941 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.849685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:44507 (local address 127.0.0.1:35728)
0810 06:25:34.850300 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.850313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.850409 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.851167 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.851170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.851188 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:34.851307 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.851314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.852704 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.852707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.854850 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.854858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.857046 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.857050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.857146 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.857168 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.857716 (+ 548us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.857799 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":120,"thread_start_us":102,"threads_started":1}
I20260810 06:25:34.860957 16389 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc1ab3eb97243b7848f68fc3815c9d6 (DEFAULT_TABLE table=test-table [id=a6769477c4b3443ea013ce110da7ff60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:34.860925 16607 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc1ab3eb97243b7848f68fc3815c9d6 (DEFAULT_TABLE table=test-table [id=a6769477c4b3443ea013ce110da7ff60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:34.861207 16172 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc1ab3eb97243b7848f68fc3815c9d6 (DEFAULT_TABLE table=test-table [id=a6769477c4b3443ea013ce110da7ff60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:34.861948 16389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc1ab3eb97243b7848f68fc3815c9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:34.861999 16607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc1ab3eb97243b7848f68fc3815c9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:34.862051 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc1ab3eb97243b7848f68fc3815c9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:34.871790 16779 tablet_bootstrap.cc:492] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Bootstrap starting.
I20260810 06:25:34.872522 16789 tablet_bootstrap.cc:492] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Bootstrap starting.
I20260810 06:25:34.873377 16788 tablet_bootstrap.cc:492] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Bootstrap starting.
I20260810 06:25:34.874789 16779 tablet_bootstrap.cc:654] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:34.875193 16789 tablet_bootstrap.cc:654] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:34.875440 16788 tablet_bootstrap.cc:654] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:34.876041 16779 log.cc:824] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:34.876281 16789 log.cc:824] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:34.876497 16788 log.cc:824] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:34.878501 16779 tablet_bootstrap.cc:492] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: No bootstrap required, opened a new log
I20260810 06:25:34.878710 16779 ts_tablet_manager.cc:1397] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:25:34.878772 16789 tablet_bootstrap.cc:492] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: No bootstrap required, opened a new log
I20260810 06:25:34.878970 16789 ts_tablet_manager.cc:1397] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260810 06:25:34.879824 16788 tablet_bootstrap.cc:492] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: No bootstrap required, opened a new log
I20260810 06:25:34.880067 16788 ts_tablet_manager.cc:1397] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 06:25:34.883841 16789 raft_consensus.cc:348] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.884086 16789 raft_consensus.cc:374] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:34.883975 16779 raft_consensus.cc:348] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.884145 16789 raft_consensus.cc:729] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1445e6450e8c4466a7f82873e98327a4, State: Initialized, Role: FOLLOWER
I20260810 06:25:34.884243 16779 raft_consensus.cc:374] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:34.884320 16779 raft_consensus.cc:729] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750304f89b244671bb6de42e1cfa4c84, State: Initialized, Role: FOLLOWER
I20260810 06:25:34.884730 16789 consensus_queue.cc:260] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.884826 16779 consensus_queue.cc:260] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.885097 16788 raft_consensus.cc:348] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.885316 16788 raft_consensus.cc:374] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:34.885360 16788 raft_consensus.cc:729] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ff81b122d944febf645f52d50d283f, State: Initialized, Role: FOLLOWER
I20260810 06:25:34.885749 16789 ts_tablet_manager.cc:1428] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260810 06:25:34.885823 16779 ts_tablet_manager.cc:1428] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:25:34.885964 16503 heartbeater.cc:503] Master 127.4.73.62:39809 was elected leader, sending a full tablet report...
I20260810 06:25:34.886472 16277 heartbeater.cc:503] Master 127.4.73.62:39809 was elected leader, sending a full tablet report...
I20260810 06:25:34.887936 16788 consensus_queue.cc:260] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.888890 16788 ts_tablet_manager.cc:1428] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260810 06:25:34.890072 16725 heartbeater.cc:503] Master 127.4.73.62:39809 was elected leader, sending a full tablet report...
I20260810 06:25:34.898284 16795 raft_consensus.cc:482] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:34.898509 16795 raft_consensus.cc:504] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.899564 16795 leader_election.cc:288] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507), 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931)
I20260810 06:25:34.903471 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.899822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:36975 (local address 127.4.73.1:44507)
0810 06:25:34.899948 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.899954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.899971 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:25:34.900174 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.900178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.900231 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.900669 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:34.900743 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.901227 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.902403 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.902839 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.902921 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.902980 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.903272 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.903357 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:25:34.903494 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.899757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:44507 (local address 127.4.73.2:36975)
0810 06:25:34.899906 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.899922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.900029 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.900369 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.900372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.900387 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:34.900533 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.900538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.901344 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.901347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.902264 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.902274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.902977 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.902984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.903129 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.903158 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.903233 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.903287 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":30}
I20260810 06:25:34.903503 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.900114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:35931 (local address 127.4.73.2:51659)
0810 06:25:34.900238 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:34.900251 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:34.900369 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:34.900627 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:34.900630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:34.900646 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:34.900764 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.900769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.901571 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.901574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.902385 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:34.902397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.903071 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.903074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:34.903192 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.903238 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 06:25:34.903317 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 06:25:34.903385 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":24}
I20260810 06:25:34.903553 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.900162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:51659 (local address 127.4.73.3:35931)
0810 06:25:34.900439 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.900442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.900453 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:25:34.900491 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.900493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.900527 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.900585 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:34.900883 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.901455 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.902515 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.902946 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.903049 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.903123 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.903355 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.903421 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260810 06:25:34.904143 16641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc1ab3eb97243b7848f68fc3815c9d6" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ff81b122d944febf645f52d50d283f" is_pre_election: true
I20260810 06:25:34.904354 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc1ab3eb97243b7848f68fc3815c9d6" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750304f89b244671bb6de42e1cfa4c84" is_pre_election: true
I20260810 06:25:34.904426 16641 raft_consensus.cc:2454] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 0.
I20260810 06:25:34.904630 16202 raft_consensus.cc:2454] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 0.
I20260810 06:25:34.904896 16321 leader_election.cc:302] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1445e6450e8c4466a7f82873e98327a4, 86ff81b122d944febf645f52d50d283f; no voters:
I20260810 06:25:34.905153 16795 raft_consensus.cc:2781] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:34.905226 16795 raft_consensus.cc:482] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:34.905254 16795 raft_consensus.cc:3037] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:34.906452 16795 raft_consensus.cc:504] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.906771 16795 leader_election.cc:288] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 election: Requested vote from peers 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507), 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931)
I20260810 06:25:34.907119 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc1ab3eb97243b7848f68fc3815c9d6" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750304f89b244671bb6de42e1cfa4c84"
I20260810 06:25:34.907292 16202 raft_consensus.cc:3037] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:34.907220 16641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc1ab3eb97243b7848f68fc3815c9d6" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ff81b122d944febf645f52d50d283f"
I20260810 06:25:34.907505 16641 raft_consensus.cc:3037] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:34.908785 16202 raft_consensus.cc:2454] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 1.
I20260810 06:25:34.909119 16321 leader_election.cc:302] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1445e6450e8c4466a7f82873e98327a4, 750304f89b244671bb6de42e1cfa4c84; no voters:
I20260810 06:25:34.909283 16795 raft_consensus.cc:2781] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:34.909564 16795 raft_consensus.cc:686] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1445e6450e8c4466a7f82873e98327a4, State: Running, Role: LEADER
I20260810 06:25:34.909889 16795 consensus_queue.cc:237] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } }
I20260810 06:25:34.910102 16641 raft_consensus.cc:2454] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 1.
I20260810 06:25:34.913743 14704 catalog_manager.cc:5495] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1445e6450e8c4466a7f82873e98327a4 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "1445e6450e8c4466a7f82873e98327a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } health_report { overall_health: UNKNOWN } } }
I20260810 06:25:34.934127 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.930512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.4.73.2:35799)
0810 06:25:34.930648 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.930653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.930666 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:34.930831 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.930834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.930896 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.930996 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:34.931245 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.931742 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.932690 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.932912 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.933016 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.933335 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:34.933735 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:34.933870 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.933932 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.933986 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:25:34.939630 16641 raft_consensus.cc:1261] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Refusing update from remote peer 1445e6450e8c4466a7f82873e98327a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:34.939815 16202 raft_consensus.cc:1261] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Refusing update from remote peer 1445e6450e8c4466a7f82873e98327a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:34.940431 16795 consensus_queue.cc:1035] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:34.940657 16799 consensus_queue.cc:1035] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:34.948729 16802 mvcc.cc:205] Tried to move back new op lower bound from 7316861480706093056 to 7316861480594583552. Current Snapshot: MvccSnapshot[applied={T|T < 7316861480706093056}]
I20260810 06:25:34.949745 16804 mvcc.cc:205] Tried to move back new op lower bound from 7316861480706093056 to 7316861480594583552. Current Snapshot: MvccSnapshot[applied={T|T < 7316861480706093056}]
I20260810 06:25:34.955550 16807 mvcc.cc:205] Tried to move back new op lower bound from 7316861480706093056 to 7316861480594583552. Current Snapshot: MvccSnapshot[applied={T|T < 7316861480706093056 or (T in {7316861480706093056})}]
I20260810 06:25:34.960847 14704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:34.961002 14704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:34.965224 14704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45105579b0a74e16b191d3ad16b5a7b7: Sending DeleteTablet for 3 replicas of tablet 2fc1ab3eb97243b7848f68fc3815c9d6
I20260810 06:25:34.966516 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc1ab3eb97243b7848f68fc3815c9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:34 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:35728
I20260810 06:25:34.966876 16389 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc1ab3eb97243b7848f68fc3815c9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:34 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:50474
I20260810 06:25:34.967130 16607 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc1ab3eb97243b7848f68fc3815c9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:34 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:53420
I20260810 06:25:34.967741 16819 tablet_replica.cc:330] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: stopping tablet replica
I20260810 06:25:34.967730 16821 tablet_replica.cc:330] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: stopping tablet replica
I20260810 06:25:34.968051 16819 raft_consensus.cc:2229] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:34.968083 16821 raft_consensus.cc:2229] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:34.968385 16819 raft_consensus.cc:2258] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:34.968581 16821 raft_consensus.cc:2258] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:34.970057 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 14656
I20260810 06:25:34.970258 16819 ts_tablet_manager.cc:1905] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:34.981015 16821 ts_tablet_manager.cc:1905] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:34.982388 16819 ts_tablet_manager.cc:1918] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:34.982499 16819 log.cc:1196] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/2fc1ab3eb97243b7848f68fc3815c9d6
I20260810 06:25:34.982839 16819 ts_tablet_manager.cc:1939] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 750304f89b244671bb6de42e1cfa4c84: Deleting consensus metadata
I20260810 06:25:34.985288 16821 ts_tablet_manager.cc:1918] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:34.985445 16821 log.cc:1196] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/2fc1ab3eb97243b7848f68fc3815c9d6
I20260810 06:25:34.985790 16821 ts_tablet_manager.cc:1939] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 86ff81b122d944febf645f52d50d283f: Deleting consensus metadata
I20260810 06:25:34.988785 16820 tablet_replica.cc:330] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: stopping tablet replica
I20260810 06:25:34.989099 16820 raft_consensus.cc:2229] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:34.989463 16820 raft_consensus.cc:2258] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:34.991108 16820 ts_tablet_manager.cc:1905] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:34.994347 16820 ts_tablet_manager.cc:1918] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:34.994443 16820 log.cc:1196] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/2fc1ab3eb97243b7848f68fc3815c9d6
I20260810 06:25:34.994752 16820 ts_tablet_manager.cc:1939] T 2fc1ab3eb97243b7848f68fc3815c9d6 P 1445e6450e8c4466a7f82873e98327a4: Deleting consensus metadata
W20260810 06:25:34.996331 16098 connection.cc:535] server connection from 127.0.0.1:35728 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 06:25:34.996328 16536 connection.cc:535] server connection from 127.0.0.1:53420 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 06:25:34.996704 16321 connection.cc:535] server connection from 127.0.0.1:50474 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:25:34.998206 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.968734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.4.73.61:33575)
0810 06:25:34.968854 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.968859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.969351 (+ 492us) server_negotiation.cc:407] Connection header received
0810 06:25:34.969542 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.969546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.969598 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.969654 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.988626 (+ 18972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.989537 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.991080 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:34.991527 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:34.991619 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:34.993940 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:34.993947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:34.993953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:34.993986 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:34.996033 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.996591 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.996595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.996597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.996697 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:34.997032 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:34.997035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:34.997037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:34.997324 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:34.997452 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:34.997835 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 06:25:34.997990 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:25:35.008678 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:34.968739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.4.73.60:39149)
0810 06:25:34.968867 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:34.968871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:34.998848 (+ 29977us) server_negotiation.cc:407] Connection header received
0810 06:25:34.999040 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:34.999043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:34.999097 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:34.999146 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:34.999981 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:35.000854 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:35.002003 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:35.002373 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:35.002461 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:35.004333 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:35.004337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:35.004339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:35.004365 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:35.006011 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:35.007048 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:35.007052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:35.007054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:35.007175 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:35.007585 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:35.007591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:35.007592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:35.007839 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:35.007949 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:35.008382 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 06:25:35.008507 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:25:35.011375 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:35.001138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.4.73.61:33575)
0810 06:25:35.001240 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:35.001244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:35.001845 (+ 601us) server_negotiation.cc:407] Connection header received
0810 06:25:35.001962 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:35.001967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:35.002016 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:35.002079 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:35.003799 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:35.004556 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:35.005766 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:35.006111 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:35.006186 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:35.007819 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:35.007823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:35.007826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:35.007848 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:35.009777 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:35.010176 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:35.010180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:35.010182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:35.010267 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:35.010564 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:35.010568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:35.010570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:35.010735 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:35.010866 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:35.011091 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 06:25:35.011200 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:35.012570 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:35.000676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.4.73.60:39149)
0810 06:25:35.000812 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:35.000816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:35.001537 (+ 721us) server_negotiation.cc:407] Connection header received
0810 06:25:35.002067 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:35.002071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:35.002115 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:35.002163 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:35.003111 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:35.003885 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:35.005289 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:35.005590 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:35.005656 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:35.007278 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:35.007282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:35.007284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:35.007310 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:35.010561 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:35.011038 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:35.011042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:35.011043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:35.011125 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:35.011597 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:35.011601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:35.011603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:35.011801 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:35.011901 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:35.012223 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 06:25:35.012386 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:25:35.953047 16848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:35.952475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:45109)
0810 06:25:35.952801 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:35.952928 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":79,"threads_started":1}
W20260810 06:25:35.953464 16277 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:39809 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:35.954684 16850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:35.954297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:56223)
0810 06:25:35.954528 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:35.954606 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
W20260810 06:25:35.954950 16503 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:39809 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:35.958022 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:35.957630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:37309)
0810 06:25:35.957844 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:35.957946 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260810 06:25:35.958303 16725 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:39809 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:36.466454 16853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 45105579b0a74e16b191d3ad16b5a7b7)
I20260810 06:25:36.466791 16854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 45105579b0a74e16b191d3ad16b5a7b7)
I20260810 06:25:36.466818 16853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:36.466977 16854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:36.468017 16853 leader_election.cc:288] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809), 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60:39149)
I20260810 06:25:36.468003 16854 leader_election.cc:288] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809), 50a46bcc18584f2a8b0ba2aa73c5ea26 (127.4.73.61:33575)
I20260810 06:25:36.468659 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.468136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51396)
0810 06:25:36.468443 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.468531 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":74,"threads_started":1}
W20260810 06:25:36.468854 15677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:36.469214 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.468390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51400)
0810 06:25:36.469048 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.469125 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":64,"threads_started":1}
W20260810 06:25:36.469388 15965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:36.469784 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.469552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51416)
0810 06:25:36.469676 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.469721 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 06:25:36.469866 15677 leader_election.cc:334] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809): Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:36.470402 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.470139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51426)
0810 06:25:36.470286 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.470335 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 06:25:36.470497 15965 leader_election.cc:334] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809): Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:36.472546 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.468163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:39149 (local address 127.0.0.1:52274)
0810 06:25:36.468477 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.468498 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:36.468608 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:36.468880 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:36.468883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:36.468903 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:36.469063 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:36.469075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.470219 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.470223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:36.471456 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:36.471467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.472180 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.472183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:36.472258 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:36.472278 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:25:36.472398 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 06:25:36.472458 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":58,"threads_started":1}
I20260810 06:25:36.472607 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.468369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.4.73.60:39149)
0810 06:25:36.468625 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:36.468631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:36.468650 (+ 19us) server_negotiation.cc:407] Connection header received
0810 06:25:36.468699 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:36.468702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:36.468760 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:36.468852 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:36.469217 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.470022 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.471587 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.472035 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.472161 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:36.472222 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:36.472416 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 06:25:36.472491 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260810 06:25:36.473124 16008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "829323b021fd4796b4dc2069177ffdc3" is_pre_election: true
I20260810 06:25:36.473302 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.468741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.4.73.61:33575)
0810 06:25:36.469230 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:36.469237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:36.469437 (+ 200us) server_negotiation.cc:407] Connection header received
0810 06:25:36.469625 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:36.469631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:36.469678 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:36.469738 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:36.470150 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.470811 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.472114 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.472720 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.472832 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:36.472890 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:36.473073 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 06:25:36.473178 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":427,"thread_start_us":60,"threads_started":1}
I20260810 06:25:36.473312 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.468940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:33575 (local address 127.0.0.1:55282)
0810 06:25:36.469359 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.469372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:36.469500 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:36.469895 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:36.469898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:36.469915 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:36.469998 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:36.470006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.470941 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.470944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:36.471991 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:36.472001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:36.472865 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:36.472869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:36.472944 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:36.472963 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 06:25:36.473053 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 06:25:36.473118 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":30,"thread_start_us":68,"threads_started":1}
I20260810 06:25:36.473373 16008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a46bcc18584f2a8b0ba2aa73c5ea26 in term 1.
I20260810 06:25:36.473688 15717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "829323b021fd4796b4dc2069177ffdc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" is_pre_election: true
I20260810 06:25:36.473733 15681 leader_election.cc:302] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50a46bcc18584f2a8b0ba2aa73c5ea26, 829323b021fd4796b4dc2069177ffdc3; no voters: 45105579b0a74e16b191d3ad16b5a7b7
I20260810 06:25:36.473873 15717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 829323b021fd4796b4dc2069177ffdc3 in term 1.
I20260810 06:25:36.473963 16853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 06:25:36.474025 16853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 45105579b0a74e16b191d3ad16b5a7b7)
I20260810 06:25:36.474071 16853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:25:36.474200 15965 leader_election.cc:302] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50a46bcc18584f2a8b0ba2aa73c5ea26, 829323b021fd4796b4dc2069177ffdc3; no voters: 45105579b0a74e16b191d3ad16b5a7b7
I20260810 06:25:36.474419 16854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 06:25:36.474486 16854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 45105579b0a74e16b191d3ad16b5a7b7)
I20260810 06:25:36.474524 16854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:25:36.475447 16853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:36.475761 16854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:36.475862 16853 leader_election.cc:288] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 2 election: Requested vote from peers 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809), 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60:39149)
I20260810 06:25:36.476140 16854 leader_election.cc:288] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [CANDIDATE]: Term 2 election: Requested vote from peers 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809), 50a46bcc18584f2a8b0ba2aa73c5ea26 (127.4.73.61:33575)
I20260810 06:25:36.476228 16008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "829323b021fd4796b4dc2069177ffdc3"
I20260810 06:25:36.476393 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.476010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51440)
0810 06:25:36.476115 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.476171 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 06:25:36.476408 16008 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 50a46bcc18584f2a8b0ba2aa73c5ea26 in current term 2: Already voted for candidate 829323b021fd4796b4dc2069177ffdc3 in this term.
I20260810 06:25:36.476562 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.476264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51444)
0810 06:25:36.476452 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.476501 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260810 06:25:36.476696 15717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "829323b021fd4796b4dc2069177ffdc3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26"
I20260810 06:25:36.476886 15717 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 829323b021fd4796b4dc2069177ffdc3 in current term 2: Already voted for candidate 50a46bcc18584f2a8b0ba2aa73c5ea26 in this term.
I20260810 06:25:36.477708 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.477549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51470)
0810 06:25:36.477621 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.477663 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 06:25:36.477854 15965 leader_election.cc:334] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809): Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:36.477924 15965 leader_election.cc:302] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 829323b021fd4796b4dc2069177ffdc3; no voters: 45105579b0a74e16b191d3ad16b5a7b7, 50a46bcc18584f2a8b0ba2aa73c5ea26
I20260810 06:25:36.478025 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.477549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51458)
0810 06:25:36.477896 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.477949 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260810 06:25:36.478060 16854 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260810 06:25:36.478240 15677 leader_election.cc:334] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809): Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:36.478322 15677 leader_election.cc:302] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50a46bcc18584f2a8b0ba2aa73c5ea26; no voters: 45105579b0a74e16b191d3ad16b5a7b7, 829323b021fd4796b4dc2069177ffdc3
I20260810 06:25:36.478475 16853 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 06:25:36.954644 16869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.954068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:38155)
0810 06:25:36.954426 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.954527 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":97,"threads_started":1}
I20260810 06:25:36.955760 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.955410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:52053)
0810 06:25:36.955637 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.955687 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260810 06:25:36.959113 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:36.958754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:55219)
0810 06:25:36.958973 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:36.959033 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260810 06:25:37.956223 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:37.955476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:47237)
0810 06:25:37.955853 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:37.955925 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":96,"threads_started":1}
I20260810 06:25:37.956766 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:37.956390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:37951)
0810 06:25:37.956639 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:37.956689 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
I20260810 06:25:37.959977 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:37.959674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:34121)
0810 06:25:37.959861 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:37.959907 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":56,"threads_started":1}
I20260810 06:25:38.067312 16885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:38.067474 16885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:38.067931 16885 leader_election.cc:288] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809), 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60:39149)
I20260810 06:25:38.068342 16008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "829323b021fd4796b4dc2069177ffdc3" is_pre_election: true
I20260810 06:25:38.068511 16008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a46bcc18584f2a8b0ba2aa73c5ea26 in term 2.
I20260810 06:25:38.068527 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.068065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51686)
0810 06:25:38.068342 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.068417 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260810 06:25:38.068847 15681 leader_election.cc:302] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50a46bcc18584f2a8b0ba2aa73c5ea26, 829323b021fd4796b4dc2069177ffdc3; no voters:
I20260810 06:25:38.069226 16885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 06:25:38.069273 16885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:38.069298 16885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:25:38.069406 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.069135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51702)
0810 06:25:38.069228 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.069297 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 06:25:38.069502 15677 leader_election.cc:334] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809): Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:38.070590 16885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:38.070916 16885 leader_election.cc:288] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 3 election: Requested vote from peers 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809), 829323b021fd4796b4dc2069177ffdc3 (127.4.73.60:39149)
I20260810 06:25:38.071233 16008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "829323b021fd4796b4dc2069177ffdc3"
I20260810 06:25:38.071296 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.071000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51712)
0810 06:25:38.071163 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.071209 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 06:25:38.071369 16008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:25:38.072278 16008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50a46bcc18584f2a8b0ba2aa73c5ea26 in term 3.
I20260810 06:25:38.072939 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.072624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51716)
0810 06:25:38.072783 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.072857 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 06:25:38.073091 15677 leader_election.cc:334] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809): Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
I20260810 06:25:38.073985 15681 leader_election.cc:302] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50a46bcc18584f2a8b0ba2aa73c5ea26, 829323b021fd4796b4dc2069177ffdc3; no voters: 45105579b0a74e16b191d3ad16b5a7b7
I20260810 06:25:38.074170 16885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 06:25:38.074383 16885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [term 3 LEADER]: Becoming Leader. State: Replica: 50a46bcc18584f2a8b0ba2aa73c5ea26, State: Running, Role: LEADER
I20260810 06:25:38.074640 16885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } }
I20260810 06:25:38.076112 16891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50a46bcc18584f2a8b0ba2aa73c5ea26. Latest consensus state: current_term: 3 leader_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:38.076220 16891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:38.076546 16893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:38.079093 16893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:38.079479 16893 catalog_manager.cc:1259] Loaded cluster ID: a798e26e8a0e4b7ea5f122b7386bd3fc
I20260810 06:25:38.079526 16893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:38.080094 16893 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:38.080621 16893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Loaded TSK: 0
I20260810 06:25:38.081274 16893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:25:38.144986 15699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:38.147964 15699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:38.151381 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.151107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51726)
0810 06:25:38.151223 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.151302 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 06:25:38.151490 16008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [term 3 FOLLOWER]: Refusing update from remote peer 50a46bcc18584f2a8b0ba2aa73c5ea26: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 06:25:38.151999 16891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 06:25:38.152119 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.151868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.0.0.1:51728)
0810 06:25:38.152002 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.152064 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 06:25:38.152282 15677 consensus_peers.cc:487] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 -> Peer 45105579b0a74e16b191d3ad16b5a7b7 (127.4.73.62:39809): Couldn't send request to peer 45105579b0a74e16b191d3ad16b5a7b7. Status: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 06:25:38.153470 16895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50a46bcc18584f2a8b0ba2aa73c5ea26. Latest consensus state: current_term: 3 leader_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:38.153611 16895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:25:38.154652 16885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:38.154889 16885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:38.155136 16895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "45105579b0a74e16b191d3ad16b5a7b7" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39809 } } peers { permanent_uuid: "50a46bcc18584f2a8b0ba2aa73c5ea26" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 33575 } attrs { promote: false } } peers { permanent_uuid: "829323b021fd4796b4dc2069177ffdc3" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 39149 } attrs { promote: false } } }
I20260810 06:25:38.155257 16895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 829323b021fd4796b4dc2069177ffdc3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:25:38.165369 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.160446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:35931 (local address 127.0.0.1:53432)
0810 06:25:38.160979 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.160995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:38.161089 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:38.161660 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:38.161663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:38.161680 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:38.161792 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:38.161799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.162898 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.162901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:38.163784 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:38.163794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.164945 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.164949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:38.165042 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:38.165063 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:25:38.165158 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:25:38.165233 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":63}
I20260810 06:25:38.165385 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.160587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.4.73.3:35931)
0810 06:25:38.161378 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:38.161384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:38.161398 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:38.161449 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:38.161457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:38.161493 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:38.161644 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:38.161898 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.162756 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.163962 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.164805 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.164947 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:38.165012 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:38.165186 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 06:25:38.165257 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":94,"threads_started":1}
I20260810 06:25:38.166214 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.161125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50492 (local address 127.4.73.2:35799)
0810 06:25:38.161363 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:38.161369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:38.162030 (+ 661us) server_negotiation.cc:407] Connection header received
0810 06:25:38.162141 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:38.162144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:38.162188 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:38.162289 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:38.162507 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.163144 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.164593 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.165215 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.165310 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:38.165380 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:38.166009 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 06:25:38.166090 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260810 06:25:38.166204 16607 tablet_service.cc:1467] Processing CreateTablet for tablet a787c08b27234c5bac69d499d98b4fb5 (DEFAULT_TABLE table=test-table [id=93ab2dc65411466790d588561f4e6403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:38.166493 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.161016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:35799 (local address 127.0.0.1:50492)
0810 06:25:38.161955 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.161970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:38.162058 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:38.162295 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:38.162298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:38.162313 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:38.162395 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:38.162401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.163272 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.163275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:38.164455 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:38.164465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.165435 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.165439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:38.165528 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:38.165549 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:25:38.165669 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 06:25:38.165738 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":84,"threads_started":1}
I20260810 06:25:38.166533 16607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a787c08b27234c5bac69d499d98b4fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:38.167071 16389 tablet_service.cc:1467] Processing CreateTablet for tablet a787c08b27234c5bac69d499d98b4fb5 (DEFAULT_TABLE table=test-table [id=93ab2dc65411466790d588561f4e6403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:38.167635 16389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a787c08b27234c5bac69d499d98b4fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:38.169294 16917 tablet_bootstrap.cc:492] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: Bootstrap starting.
I20260810 06:25:38.170171 16917 tablet_bootstrap.cc:654] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:38.170230 16918 tablet_bootstrap.cc:492] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: Bootstrap starting.
I20260810 06:25:38.171327 16918 tablet_bootstrap.cc:654] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:38.171727 16917 tablet_bootstrap.cc:492] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: No bootstrap required, opened a new log
I20260810 06:25:38.171813 16917 ts_tablet_manager.cc:1397] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:25:38.172191 16917 raft_consensus.cc:348] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.172333 16917 raft_consensus.cc:374] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:38.172372 16917 raft_consensus.cc:729] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ff81b122d944febf645f52d50d283f, State: Initialized, Role: FOLLOWER
I20260810 06:25:38.172485 16917 consensus_queue.cc:260] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.172813 16917 ts_tablet_manager.cc:1428] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:38.173364 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.173114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:49833)
0810 06:25:38.173238 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.173294 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 06:25:38.174204 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.173827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:41443)
0810 06:25:38.174068 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.174140 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 06:25:38.175122 16726 heartbeater.cc:503] Master 127.4.73.61:33575 was elected leader, sending a full tablet report...
I20260810 06:25:38.176244 16918 tablet_bootstrap.cc:492] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: No bootstrap required, opened a new log
I20260810 06:25:38.176374 16918 ts_tablet_manager.cc:1397] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 06:25:38.176813 16918 raft_consensus.cc:348] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.176946 16918 raft_consensus.cc:374] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:38.176987 16918 raft_consensus.cc:729] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1445e6450e8c4466a7f82873e98327a4, State: Initialized, Role: FOLLOWER
I20260810 06:25:38.177011 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.160692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:44507 (local address 127.0.0.1:35738)
0810 06:25:38.161098 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.161111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:38.161180 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:38.161761 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:38.161764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:38.161784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:38.161879 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:38.161885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.162879 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.162882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:38.175360 (+ 12478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:38.175373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.176589 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.176593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:38.176694 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:38.176713 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 06:25:38.176803 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 06:25:38.176885 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":110,"threads_started":1}
I20260810 06:25:38.177048 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.160832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.4.73.1:44507)
0810 06:25:38.161399 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:38.161404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:38.161418 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:38.161471 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:38.161473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:38.161521 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:38.161614 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:38.161988 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.162756 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.175529 (+ 12773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.176242 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.176452 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:38.176532 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:38.176837 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 06:25:38.176921 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":106,"threads_started":1}
I20260810 06:25:38.177155 16918 consensus_queue.cc:260] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.177533 16918 ts_tablet_manager.cc:1428] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:38.177749 16172 tablet_service.cc:1467] Processing CreateTablet for tablet a787c08b27234c5bac69d499d98b4fb5 (DEFAULT_TABLE table=test-table [id=93ab2dc65411466790d588561f4e6403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:38.178046 16172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a787c08b27234c5bac69d499d98b4fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:38.178287 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.177941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:42501)
0810 06:25:38.178164 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.178222 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260810 06:25:38.179235 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.178988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:52857)
0810 06:25:38.179109 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.179162 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 06:25:38.179327 16505 heartbeater.cc:503] Master 127.4.73.61:33575 was elected leader, sending a full tablet report...
I20260810 06:25:38.180917 16924 tablet_bootstrap.cc:492] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84: Bootstrap starting.
I20260810 06:25:38.181928 16924 tablet_bootstrap.cc:654] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:38.183017 16924 tablet_bootstrap.cc:492] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84: No bootstrap required, opened a new log
I20260810 06:25:38.183089 16924 ts_tablet_manager.cc:1397] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:38.183431 16924 raft_consensus.cc:348] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.183516 16924 raft_consensus.cc:374] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:38.183538 16924 raft_consensus.cc:729] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750304f89b244671bb6de42e1cfa4c84, State: Initialized, Role: FOLLOWER
I20260810 06:25:38.183629 16924 consensus_queue.cc:260] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.183889 16924 ts_tablet_manager.cc:1428] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:38.184537 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.184259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:50239)
0810 06:25:38.184410 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.184466 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 06:25:38.185647 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.184973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:59349)
0810 06:25:38.185506 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.185573 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465}
I20260810 06:25:38.238384 16923 raft_consensus.cc:482] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:38.238551 16923 raft_consensus.cc:504] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.238991 16923 leader_election.cc:288] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931), 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507)
I20260810 06:25:38.239457 16641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a787c08b27234c5bac69d499d98b4fb5" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ff81b122d944febf645f52d50d283f" is_pre_election: true
I20260810 06:25:38.239506 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a787c08b27234c5bac69d499d98b4fb5" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750304f89b244671bb6de42e1cfa4c84" is_pre_election: true
I20260810 06:25:38.239625 16641 raft_consensus.cc:2454] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 0.
I20260810 06:25:38.239666 16202 raft_consensus.cc:2454] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 0.
I20260810 06:25:38.239992 16321 leader_election.cc:302] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1445e6450e8c4466a7f82873e98327a4, 750304f89b244671bb6de42e1cfa4c84; no voters:
I20260810 06:25:38.240183 16923 raft_consensus.cc:2781] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:38.240243 16923 raft_consensus.cc:482] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:38.240288 16923 raft_consensus.cc:3037] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:38.241336 16923 raft_consensus.cc:504] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.241677 16923 leader_election.cc:288] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 election: Requested vote from peers 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931), 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507)
I20260810 06:25:38.242000 16641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a787c08b27234c5bac69d499d98b4fb5" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ff81b122d944febf645f52d50d283f"
I20260810 06:25:38.242103 16641 raft_consensus.cc:3037] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:38.242101 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a787c08b27234c5bac69d499d98b4fb5" candidate_uuid: "1445e6450e8c4466a7f82873e98327a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750304f89b244671bb6de42e1cfa4c84"
I20260810 06:25:38.242241 16202 raft_consensus.cc:3037] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:38.243045 16641 raft_consensus.cc:2454] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 1.
I20260810 06:25:38.243208 16202 raft_consensus.cc:2454] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1445e6450e8c4466a7f82873e98327a4 in term 1.
I20260810 06:25:38.243356 16321 leader_election.cc:302] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1445e6450e8c4466a7f82873e98327a4, 86ff81b122d944febf645f52d50d283f; no voters:
I20260810 06:25:38.243578 16923 raft_consensus.cc:2781] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:38.243806 16923 raft_consensus.cc:686] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1445e6450e8c4466a7f82873e98327a4, State: Running, Role: LEADER
I20260810 06:25:38.243964 16923 consensus_queue.cc:237] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } }
I20260810 06:25:38.244557 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.244263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:40445)
0810 06:25:38.244422 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.244484 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 06:25:38.245747 15699 catalog_manager.cc:5495] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1445e6450e8c4466a7f82873e98327a4 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "1445e6450e8c4466a7f82873e98327a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1445e6450e8c4466a7f82873e98327a4" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35799 } health_report { overall_health: HEALTHY } } }
I20260810 06:25:38.260897 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.258239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.4.73.2:35799)
0810 06:25:38.258381 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:38.258385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:38.258397 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:25:38.258435 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:38.258437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:38.258476 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:38.258540 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:38.258745 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.259233 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.260009 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:38.260205 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:38.260276 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:38.260541 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:38.260605 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:38.260659 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:38.260754 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 06:25:38.260797 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:25:38.263221 16202 raft_consensus.cc:1261] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Refusing update from remote peer 1445e6450e8c4466a7f82873e98327a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:38.263695 16641 raft_consensus.cc:1261] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Refusing update from remote peer 1445e6450e8c4466a7f82873e98327a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:38.264169 16923 consensus_queue.cc:1035] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750304f89b244671bb6de42e1cfa4c84" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:38.264340 16931 consensus_queue.cc:1035] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ff81b122d944febf645f52d50d283f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 35931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:38.266175 16278 heartbeater.cc:503] Master 127.4.73.61:33575 was elected leader, sending a full tablet report...
I20260810 06:25:38.267076 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.266839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:39851)
0810 06:25:38.266938 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.267002 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 06:25:38.270036 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.269778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:38795)
0810 06:25:38.269914 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.269968 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 06:25:38.272588 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.272192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:46893)
0810 06:25:38.272458 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.272512 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260810 06:25:38.273440 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.273118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.1:42233)
0810 06:25:38.273301 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.273354 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260810 06:25:38.274885 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.274614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.2:34623)
0810 06:25:38.274742 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.274819 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 06:25:38.276695 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.276443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39809 (local address 127.4.73.3:48225)
0810 06:25:38.276578 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.276630 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:39809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 06:25:38.277951 15699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:38.278103 15699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:38.281461 15699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50a46bcc18584f2a8b0ba2aa73c5ea26: Sending DeleteTablet for 3 replicas of tablet a787c08b27234c5bac69d499d98b4fb5
I20260810 06:25:38.282209 16389 tablet_service.cc:1514] Processing DeleteTablet for tablet a787c08b27234c5bac69d499d98b4fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:38 UTC) from {username='oryx', principal='oryx/127.4.73.61@KRBTEST.COM'} at 127.0.0.1:50492
I20260810 06:25:38.282310 16607 tablet_service.cc:1514] Processing DeleteTablet for tablet a787c08b27234c5bac69d499d98b4fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:38 UTC) from {username='oryx', principal='oryx/127.4.73.61@KRBTEST.COM'} at 127.0.0.1:53432
I20260810 06:25:38.282423 16172 tablet_service.cc:1514] Processing DeleteTablet for tablet a787c08b27234c5bac69d499d98b4fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:38 UTC) from {username='oryx', principal='oryx/127.4.73.61@KRBTEST.COM'} at 127.0.0.1:35738
I20260810 06:25:38.282544 16954 tablet_replica.cc:330] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: stopping tablet replica
I20260810 06:25:38.282658 16954 raft_consensus.cc:2229] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:38.282676 16955 tablet_replica.cc:330] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: stopping tablet replica
I20260810 06:25:38.282812 16955 raft_consensus.cc:2229] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:38.282858 16954 raft_consensus.cc:2258] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:38.282951 16955 raft_consensus.cc:2258] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:38.283022 16957 tablet_replica.cc:330] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84: stopping tablet replica
I20260810 06:25:38.283262 16957 raft_consensus.cc:2229] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:38.283399 16957 raft_consensus.cc:2258] T a787c08b27234c5bac69d499d98b4fb5 P 750304f89b244671bb6de42e1cfa4c84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:38.283605 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 15935
I20260810 06:25:38.284197 16954 ts_tablet_manager.cc:1905] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:38.286330 16954 ts_tablet_manager.cc:1918] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:38.286435 16954 log.cc:1196] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/a787c08b27234c5bac69d499d98b4fb5
I20260810 06:25:38.286729 16954 ts_tablet_manager.cc:1939] T a787c08b27234c5bac69d499d98b4fb5 P 1445e6450e8c4466a7f82873e98327a4: Deleting consensus metadata
I20260810 06:25:38.287865 15681 catalog_manager.cc:4841] TS 1445e6450e8c4466a7f82873e98327a4 (127.4.73.2:35799): tablet a787c08b27234c5bac69d499d98b4fb5 (table test-table [id=93ab2dc65411466790d588561f4e6403]) successfully deleted
I20260810 06:25:38.296962 16955 ts_tablet_manager.cc:1905] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:38.298287 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 16291
I20260810 06:25:38.303514 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:38.298583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:44507 (local address 127.0.0.1:35744)
0810 06:25:38.303285 (+ 4702us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:38.303384 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:44507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4581}
W20260810 06:25:38.304656 15678 catalog_manager.cc:4584] TS 750304f89b244671bb6de42e1cfa4c84 (127.4.73.1:44507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a787c08b27234c5bac69d499d98b4fb5: Network error: Client connection negotiation failed: client connection to 127.4.73.1:44507: connect: Connection refused (error 111)
I20260810 06:25:38.305872 16955 ts_tablet_manager.cc:1918] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:38.305953 16955 log.cc:1196] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/a787c08b27234c5bac69d499d98b4fb5
I20260810 06:25:38.306205 16955 ts_tablet_manager.cc:1939] T a787c08b27234c5bac69d499d98b4fb5 P 86ff81b122d944febf645f52d50d283f: Deleting consensus metadata
I20260810 06:25:38.307070 15678 catalog_manager.cc:4841] TS 86ff81b122d944febf645f52d50d283f (127.4.73.3:35931): tablet a787c08b27234c5bac69d499d98b4fb5 (table test-table [id=93ab2dc65411466790d588561f4e6403]) successfully deleted
I20260810 06:25:38.315426 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 16513
I20260810 06:25:38.329500 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 15467
I20260810 06:25:38.341391 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 15645
2026-08-10T06:25:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7140 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:38 dist-test-slave-45dx krb5kdc[16961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:38 dist-test-slave-45dx krb5kdc[16961](info): set up 2 sockets
Aug 10 06:25:38 dist-test-slave-45dx krb5kdc[16961](info): commencing operation
krb5kdc: starting...
W20260810 06:25:39.621882 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.251s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:39 dist-test-slave-45dx krb5kdc[16961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab.
Entry for principal oryx/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab.
I20260810 06:25:39.745422 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:35091
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab
--principal=oryx/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:35091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:39.852123 16982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:39.852445 16982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:39.852546 16982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:39.856230 16982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:39.856314 16982 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 06:25:39.856364 16982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:39.856398 16982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:39.856433 16982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:39.856470 16982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:39.861600 16982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:35091
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.62.keytab
--principal=oryx/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:35091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:39.862619 16982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:39.863602 16982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:39.872107 16990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:39.872120 16995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:39.872156 16991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:39.872579 16982 server_base.cc:1034] running on GCE node
I20260810 06:25:39.873170 16982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:39.874215 16982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:39.875381 16982 hybrid_clock.cc:648] HybridClock initialized: now 1786343139875361 us; error 32 us; skew 500 ppm
Aug 10 06:25:39 dist-test-slave-45dx krb5kdc[16961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343139, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:39.878147 16982 init.cc:377] Logged in from keytab as oryx/127.4.73.62@KRBTEST.COM (short username oryx)
I20260810 06:25:39.879024 16982 webserver.cc:460] Webserver started at http://127.4.73.62:45751/ using document root <none> and password file <none>
I20260810 06:25:39.879552 16982 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:39.879640 16982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:39.879873 16982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:39.881865 16982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "cdd18b25307c4dba9c9e6c89cfa48144"
format_stamp: "Formatted at 2026-08-10 06:25:39 on dist-test-slave-45dx"
server_key: "5e46041525ac18d65db2f7157a2ea437"
server_key_iv: "d6dcc43e415471ab58b4e879e47b12dc"
server_key_version: "encryptionkey@0"
I20260810 06:25:39.882429 16982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "cdd18b25307c4dba9c9e6c89cfa48144"
format_stamp: "Formatted at 2026-08-10 06:25:39 on dist-test-slave-45dx"
server_key: "5e46041525ac18d65db2f7157a2ea437"
server_key_iv: "d6dcc43e415471ab58b4e879e47b12dc"
server_key_version: "encryptionkey@0"
I20260810 06:25:39.885938 16982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 06:25:39.888160 17006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:39.889396 16982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260810 06:25:39.889560 16982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "cdd18b25307c4dba9c9e6c89cfa48144"
format_stamp: "Formatted at 2026-08-10 06:25:39 on dist-test-slave-45dx"
server_key: "5e46041525ac18d65db2f7157a2ea437"
server_key_iv: "d6dcc43e415471ab58b4e879e47b12dc"
server_key_version: "encryptionkey@0"
I20260810 06:25:39.889672 16982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:39.912909 16982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:39.915587 16982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:39.915798 16982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:39.923657 16982 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:35091
I20260810 06:25:39.923686 17115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:35091 every 8 connection(s)
I20260810 06:25:39.924568 16982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:39.927104 17117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:39.931332 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 16982
I20260810 06:25:39.931427 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:39.931726 4388 external_mini_cluster.cc:1428] Setting key 746c2e3f0f8632fc7798dd3f50048e1d
I20260810 06:25:39.932560 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Bootstrap starting.
I20260810 06:25:39.934888 17117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:39.935632 17117 log.cc:824] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:39.937464 17117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: No bootstrap required, opened a new log
Aug 10 06:25:39 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:39.941454 17117 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35091 } }
I20260810 06:25:39.941730 17117 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:39.941859 17117 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd18b25307c4dba9c9e6c89cfa48144, State: Initialized, Role: FOLLOWER
I20260810 06:25:39.942385 17117 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35091 } }
I20260810 06:25:39.942551 17117 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:39.942662 17117 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:39.942770 17117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:39.943785 17117 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35091 } }
I20260810 06:25:39.944123 17117 leader_election.cc:302] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdd18b25307c4dba9c9e6c89cfa48144; no voters:
I20260810 06:25:39.944470 17117 leader_election.cc:288] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:39.944531 17124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:39.944742 17124 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [term 1 LEADER]: Becoming Leader. State: Replica: cdd18b25307c4dba9c9e6c89cfa48144, State: Running, Role: LEADER
I20260810 06:25:39.945014 17124 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35091 } }
I20260810 06:25:39.945693 17117 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:39.946062 17126 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35091 } } }
I20260810 06:25:39.946172 17126 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:39.946331 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:39.933125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.4.73.62:35091)
0810 06:25:39.933570 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:39.933584 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:39.933618 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:25:39.934249 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:39.934265 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:39.934601 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:39.934931 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:39.935765 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:39.936820 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:39.937690 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:39.938153 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:39.938319 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:39.941129 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:39.941134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:39.941145 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:39.941169 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:39.943214 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:39.943694 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:39.943699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:39.943703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:39.943829 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:39.944116 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:39.944120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:39.944121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:39.944624 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:39.944828 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:39.945159 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 06:25:39.945472 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":138,"threads_started":1}
I20260810 06:25:39.946400 17125 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdd18b25307c4dba9c9e6c89cfa48144. Latest consensus state: current_term: 1 leader_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdd18b25307c4dba9c9e6c89cfa48144" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 35091 } } }
I20260810 06:25:39.946805 17128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:39.946471 17125 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:39.950016 17128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:39.955142 17128 catalog_manager.cc:1347] Generated new cluster ID: 26fe4db6e9a24dd0b1ae1b37ff2314d7
I20260810 06:25:39.955209 17128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:39.979326 17128 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:39.980268 17128 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:39.984689 17128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Generated new TSK 0
I20260810 06:25:39.985307 17128 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab.
Entry for principal oryx/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab.
I20260810 06:25:40.019060 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:35091
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab
--principal=oryx/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:40.161177 17155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:40.161414 17155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:40.161490 17155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:40.165227 17155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:40.165272 17155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:40.165359 17155 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:40.165446 17155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:40.170364 17155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab
--principal=oryx/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:40.171314 17155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:40.172367 17155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:40.181726 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:40.181734 17166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:40.181803 17155 server_base.cc:1034] running on GCE node
W20260810 06:25:40.181734 17164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:40.182334 17155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:40.182967 17155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:40.184161 17155 hybrid_clock.cc:648] HybridClock initialized: now 1786343140184144 us; error 30 us; skew 500 ppm
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:40.186968 17155 init.cc:377] Logged in from keytab as oryx/127.4.73.1@KRBTEST.COM (short username oryx)
I20260810 06:25:40.187827 17155 webserver.cc:460] Webserver started at http://127.4.73.1:34769/ using document root <none> and password file <none>
I20260810 06:25:40.188455 17155 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:40.188555 17155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:40.188792 17155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:40.190830 17155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "6fe9d84a2d9749fb8dd95664f389a48c"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "0f7133536a8be92355a150553faa777a"
server_key_iv: "38adf8977fa15391825c7003ae1e0a49"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.191421 17155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "6fe9d84a2d9749fb8dd95664f389a48c"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "0f7133536a8be92355a150553faa777a"
server_key_iv: "38adf8977fa15391825c7003ae1e0a49"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.195308 17155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 06:25:40.197721 17183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:40.198968 17155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:40.199121 17155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "6fe9d84a2d9749fb8dd95664f389a48c"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "0f7133536a8be92355a150553faa777a"
server_key_iv: "38adf8977fa15391825c7003ae1e0a49"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.199239 17155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:40.226151 17155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:40.229143 17155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:40.229372 17155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:40.230886 17155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:40.230970 17155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:40.231047 17155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:40.231081 17155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:40.242462 17155 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:43073
I20260810 06:25:40.242492 17391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:43073 every 8 connection(s)
I20260810 06:25:40.243535 17155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:40.245814 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.231810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:58315 (local address 127.4.73.62:35091)
0810 06:25:40.231941 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.231944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.233020 (+ 1076us) server_negotiation.cc:407] Connection header received
0810 06:25:40.234096 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.234101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.234187 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.234255 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.236077 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.237087 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.238514 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.238923 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.239062 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.242091 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:40.242095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:40.242098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:40.242129 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:40.243812 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.244441 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.244444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.244446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.244541 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.244874 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.244876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.244878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.245043 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:40.245113 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.245507 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.245604 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:25:40.246500 17199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.232401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:35091 (local address 127.4.73.1:58315)
0810 06:25:40.232874 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.232915 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.233812 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.234715 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.234724 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.235168 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:40.235894 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.235913 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.237236 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.237240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.238383 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.238393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.239102 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.239107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.239296 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.239873 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:40.239893 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:40.241899 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:40.243963 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.243970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.243982 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.244327 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.244639 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.244641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.244642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.244766 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.245123 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:40.245127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:40.245319 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.245447 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.245691 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260810 06:25:40.247921 17392 heartbeater.cc:347] Connected to a master server at 127.4.73.62:35091
I20260810 06:25:40.248075 17392 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:40.248629 17392 heartbeater.cc:511] Master 127.4.73.62:35091 requested a full tablet report, sending...
I20260810 06:25:40.250078 17033 ts_manager.cc:194] Registered new tserver with Master: 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073)
I20260810 06:25:40.251171 17033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:58315
I20260810 06:25:40.251549 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 17155
I20260810 06:25:40.251675 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:40.252007 4388 external_mini_cluster.cc:1428] Setting key 255b197940a1c3097f8b7a7f15805d50
I20260810 06:25:40.257539 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.250174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:42525 (local address 127.4.73.62:35091)
0810 06:25:40.250288 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.250292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.250503 (+ 211us) server_negotiation.cc:407] Connection header received
0810 06:25:40.250613 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.250616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.250655 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.250745 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.251158 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.251887 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.252859 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.253076 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.253182 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.254438 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:40.254442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:40.254444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:40.254467 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:40.255827 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.256295 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.256298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.256334 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.256423 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.256706 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.256710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.256712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.256902 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:40.257011 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.257265 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.257375 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:25:40.257557 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.250066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:35091 (local address 127.4.73.1:42525)
0810 06:25:40.250419 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.250430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.250540 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.250758 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.250761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.250957 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:40.251048 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.251053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.252016 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.252018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.252724 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.252736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.253188 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.253191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.253294 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.253949 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:40.253964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:40.254309 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:40.255977 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.255982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.255984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.256186 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.256520 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.256524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.256525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.256613 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.257001 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:40.257005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:40.257105 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.257212 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.257371 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab.
Entry for principal oryx/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab.
I20260810 06:25:40.282547 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:35091
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab
--principal=oryx/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:40.393795 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:40.394068 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:40.394129 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:40.398047 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:40.398099 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:40.398137 17406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:40.398168 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:40.403313 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab
--principal=oryx/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:40.404299 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:40.405318 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:40.415079 17423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:40.415096 17415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:40.415079 17414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:40.415279 17406 server_base.cc:1034] running on GCE node
I20260810 06:25:40.415848 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:40.416500 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:40.417663 17406 hybrid_clock.cc:648] HybridClock initialized: now 1786343140417631 us; error 44 us; skew 500 ppm
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:40.420544 17406 init.cc:377] Logged in from keytab as oryx/127.4.73.2@KRBTEST.COM (short username oryx)
I20260810 06:25:40.421443 17406 webserver.cc:460] Webserver started at http://127.4.73.2:46009/ using document root <none> and password file <none>
I20260810 06:25:40.422062 17406 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:40.422158 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:40.422397 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:40.424429 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "28337dc5aec943249975e6eb051466f0"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "0febd5c85276033ec97773b4be6002f4"
server_key_iv: "ae3e3debc42c29c01c1fccb5ad93ac3e"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.425001 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "28337dc5aec943249975e6eb051466f0"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "0febd5c85276033ec97773b4be6002f4"
server_key_iv: "ae3e3debc42c29c01c1fccb5ad93ac3e"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.428512 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:25:40.430716 17431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:40.431834 17406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:40.431977 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "28337dc5aec943249975e6eb051466f0"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "0febd5c85276033ec97773b4be6002f4"
server_key_iv: "ae3e3debc42c29c01c1fccb5ad93ac3e"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.432082 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:40.444373 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:40.447623 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:40.447829 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:40.449460 17406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:40.449529 17406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:40.449608 17406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:40.449632 17406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:40.460448 17406 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:42331
I20260810 06:25:40.460683 17625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:42331 every 8 connection(s)
I20260810 06:25:40.461328 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:40.463016 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.450082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:39125 (local address 127.4.73.62:35091)
0810 06:25:40.450205 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.450209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.451185 (+ 976us) server_negotiation.cc:407] Connection header received
0810 06:25:40.452152 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.452156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.452215 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.452320 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.453651 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.454562 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.455729 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.455955 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.456067 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.459382 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:40.459386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:40.459389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:40.459422 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:40.461018 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.461550 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.461554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.461556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.461654 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.462080 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.462083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.462084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.462283 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:40.462365 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.462789 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.462899 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:40.463758 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.450572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:35091 (local address 127.4.73.2:39125)
0810 06:25:40.451025 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.451062 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.451951 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.452641 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.452649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.452950 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:40.453499 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.453513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.454706 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.454712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.455573 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.455583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.456128 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.456133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.456371 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.457013 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:40.457036 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:40.459191 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:40.461136 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.461141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.461155 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.461451 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.461759 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.461762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.461764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.461961 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.462364 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:40.462369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:40.462577 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.462741 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.462977 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":136,"threads_started":1}
I20260810 06:25:40.465073 17627 heartbeater.cc:347] Connected to a master server at 127.4.73.62:35091
I20260810 06:25:40.465227 17627 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:40.465716 17627 heartbeater.cc:511] Master 127.4.73.62:35091 requested a full tablet report, sending...
I20260810 06:25:40.466747 17035 ts_manager.cc:194] Registered new tserver with Master: 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331)
I20260810 06:25:40.467541 17035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:39125
I20260810 06:25:40.469081 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 17406
I20260810 06:25:40.469195 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:40.469501 4388 external_mini_cluster.cc:1428] Setting key 25c1ffe2785c2914e35d599e944a28de
I20260810 06:25:40.474586 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.467130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:35091 (local address 127.4.73.2:42565)
0810 06:25:40.467440 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.467453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.467543 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.467840 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.467843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.468051 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:40.468174 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.468181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.469048 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.469051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.469843 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.469863 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.470373 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.470378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.470472 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.470987 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:40.470999 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:40.471315 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:40.473117 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.473120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.473122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.473327 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.473635 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.473637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.473638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.473714 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.474200 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:40.474202 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:40.474271 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.474359 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.474463 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260810 06:25:40.474771 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.467205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:42565 (local address 127.4.73.62:35091)
0810 06:25:40.467316 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.467319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.467522 (+ 203us) server_negotiation.cc:407] Connection header received
0810 06:25:40.467649 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.467654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.467726 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.467789 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.468294 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.468877 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.469977 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.470250 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.470334 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.471449 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:40.471453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:40.471456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:40.471478 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:40.472984 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.473440 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.473445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.473449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.473542 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.473803 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.473809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.473810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.474102 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:40.474214 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.474445 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.474575 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab.
Entry for principal oryx/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab.
I20260810 06:25:40.499336 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:35091
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab
--principal=oryx/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:40.619951 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:40.620190 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:40.620234 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:40.624452 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:40.624513 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:40.624552 17640 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 06:25:40.624625 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:40.629818 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab
--principal=oryx/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.73.62:35091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:40.630818 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:40.631834 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:40.641434 17651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:40.641407 17647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:40.641606 17640 server_base.cc:1034] running on GCE node
W20260810 06:25:40.641415 17649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:40.642136 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:40.642673 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:40.643836 17640 hybrid_clock.cc:648] HybridClock initialized: now 1786343140643817 us; error 30 us; skew 500 ppm
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:40.646678 17640 init.cc:377] Logged in from keytab as oryx/127.4.73.3@KRBTEST.COM (short username oryx)
I20260810 06:25:40.647632 17640 webserver.cc:460] Webserver started at http://127.4.73.3:40885/ using document root <none> and password file <none>
I20260810 06:25:40.648188 17640 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:40.648267 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:40.648517 17640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:40.650538 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "a4640e7d83194665ae03ce5f717bca8b"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "1f1f8939cf36c8a83a970d4d04f45f29"
server_key_iv: "13cb8633972e64c927903833fca1fc42"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.651108 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "a4640e7d83194665ae03ce5f717bca8b"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "1f1f8939cf36c8a83a970d4d04f45f29"
server_key_iv: "13cb8633972e64c927903833fca1fc42"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.654703 17640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:25:40.657042 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:40.658187 17640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:40.658326 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "a4640e7d83194665ae03ce5f717bca8b"
format_stamp: "Formatted at 2026-08-10 06:25:40 on dist-test-slave-45dx"
server_key: "1f1f8939cf36c8a83a970d4d04f45f29"
server_key_iv: "13cb8633972e64c927903833fca1fc42"
server_key_version: "encryptionkey@0"
I20260810 06:25:40.658425 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:40.684195 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:40.686952 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:40.687150 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:40.688606 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:40.688656 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:40.688721 17640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:40.688755 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:40.700121 17640 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:44827
I20260810 06:25:40.700984 17849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:44827 every 8 connection(s)
I20260810 06:25:40.701185 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:40.703073 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.689275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:56633 (local address 127.4.73.62:35091)
0810 06:25:40.689440 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.689445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.690307 (+ 862us) server_negotiation.cc:407] Connection header received
0810 06:25:40.691568 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.691572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.691627 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.691717 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.693170 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.693977 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.695343 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.695668 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.695796 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.699124 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:40.699127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:40.699129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:40.699155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:40.700835 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.701456 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.701459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.701460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.701538 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.701956 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.701959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.701961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.702202 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:40.702308 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.702782 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.702921 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 06:25:40.703840 17677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.689633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:35091 (local address 127.4.73.3:56633)
0810 06:25:40.690121 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.690181 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.691323 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.691922 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.691934 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.692464 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:40.693025 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.693040 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.694103 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.694106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.695216 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.695227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.695842 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.695847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.696043 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.696685 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:40.696708 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:40.698942 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:40.700962 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.700967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.700980 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.701358 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.701641 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.701644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.701646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.701833 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.702331 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:40.702336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:40.702537 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.702690 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.703093 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
I20260810 06:25:40.704995 17851 heartbeater.cc:347] Connected to a master server at 127.4.73.62:35091
I20260810 06:25:40.705154 17851 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:40.705642 17851 heartbeater.cc:511] Master 127.4.73.62:35091 requested a full tablet report, sending...
I20260810 06:25:40.706635 17033 ts_manager.cc:194] Registered new tserver with Master: a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827)
I20260810 06:25:40.707093 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 17640
I20260810 06:25:40.707239 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:40.707376 17033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:56633
I20260810 06:25:40.707625 4388 external_mini_cluster.cc:1428] Setting key 3535a313e51ce28210bd27672ede7503
I20260810 06:25:40.709564 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:25:40.715122 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.707650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:52455 (local address 127.4.73.62:35091)
0810 06:25:40.707777 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.707781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.708186 (+ 405us) server_negotiation.cc:407] Connection header received
0810 06:25:40.708282 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.708285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.708352 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.708438 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.709005 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.709686 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.710504 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.710894 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.711014 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.712050 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:40.712053 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:40.712056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:40.712080 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:40.713520 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.713950 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.713953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.713955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.714041 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.714326 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.714330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.714331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.714497 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:40.714605 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.714871 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.714975 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 06:25:40.715129 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.707042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:35091 (local address 127.4.73.3:52455)
0810 06:25:40.708092 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.708107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.708202 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.708542 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.708545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.708749 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:40.708868 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.708875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.709845 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.709847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.710389 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.710396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.711013 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.711015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.711084 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.711583 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:40.711595 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:40.711908 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:40.713637 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.713641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.713643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.713847 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.714134 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:40.714138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:40.714140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:40.714227 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:40.714597 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:40.714600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:40.714698 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.714811 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.714947 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":93,"threads_started":1}
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:25:40.734342 17122 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:40.728656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.4.73.62:35091)
0810 06:25:40.728779 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.728782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.728794 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:25:40.728828 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.728830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.728866 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.728937 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.729659 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.730145 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.730967 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.731167 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.731240 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.734213 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59768: BlockingRecv error: recv got EOF from 127.0.0.1:59768 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 06:25:40 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343140, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:40.743330 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.735914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.4.73.62:35091)
0810 06:25:40.736035 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.736039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.736178 (+ 139us) server_negotiation.cc:407] Connection header received
0810 06:25:40.736258 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.736260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.736292 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.736414 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:40.737030 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.737494 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.738221 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.738420 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.738492 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.740553 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:40.740555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:40.740557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:40.740579 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:40.742005 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.742357 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.742360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.742361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.742434 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:40.742659 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:40.742662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:40.742663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:40.742810 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:40.742909 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.743070 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.743151 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:25:40.746155 17033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:40.749289 17033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:40.762284 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.757036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:44827 (local address 127.0.0.1:55048)
0810 06:25:40.757382 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.757431 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.757552 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.758320 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.758326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.758362 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:40.758482 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.758490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.759646 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.759652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.760691 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.760705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.761761 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.761765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.761883 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.761919 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.762039 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.762156 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260810 06:25:40.762298 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.757192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.4.73.3:44827)
0810 06:25:40.757524 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.757536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.757562 (+ 26us) server_negotiation.cc:407] Connection header received
0810 06:25:40.757746 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.757753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.757970 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.758129 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:40.758626 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.759525 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.760844 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.761621 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.761786 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.761958 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.762069 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.762161 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":115,"threads_started":1}
I20260810 06:25:40.763538 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.757754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:42331 (local address 127.0.0.1:47786)
0810 06:25:40.758469 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.758482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.758572 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.759184 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.759187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.759206 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:40.759293 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.759298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.760801 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.760805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.762065 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.762077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.763169 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.763173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.763260 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.763280 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.763347 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.763418 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":96,"threads_started":1}
I20260810 06:25:40.763595 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.758062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.4.73.2:42331)
0810 06:25:40.758730 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.758741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.758762 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:25:40.758843 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.758848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.759032 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.759187 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:40.759413 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.760668 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.762221 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.763044 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.763166 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.763320 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.763384 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.763465 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":105,"threads_started":1}
I20260810 06:25:40.764202 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.758088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:43073 (local address 127.0.0.1:60832)
0810 06:25:40.758758 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.758771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.758858 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.759505 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.759507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.759525 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:40.759619 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.759625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.761232 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.761236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.762685 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.762696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.763815 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.763819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.763922 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.763949 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.764011 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.764077 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":92,"threads_started":1}
I20260810 06:25:40.765054 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.758480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.4.73.1:43073)
0810 06:25:40.759071 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.759078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.759103 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:25:40.759177 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.759182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.759372 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.759527 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:40.759728 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.760837 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.762797 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.763562 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.764644 (+ 1082us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.764778 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.764828 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.764927 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":93,"threads_started":1}
I20260810 06:25:40.765550 17742 tablet_service.cc:1467] Processing CreateTablet for tablet a049161e3aef4cd39b67be457d116c71 (DEFAULT_TABLE table=test-table [id=25654ce7dc8f4d42bddfd929ef50d487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:40.766554 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a049161e3aef4cd39b67be457d116c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:40.766299 17510 tablet_service.cc:1467] Processing CreateTablet for tablet a049161e3aef4cd39b67be457d116c71 (DEFAULT_TABLE table=test-table [id=25654ce7dc8f4d42bddfd929ef50d487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:40.767261 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a049161e3aef4cd39b67be457d116c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:40.767588 17269 tablet_service.cc:1467] Processing CreateTablet for tablet a049161e3aef4cd39b67be457d116c71 (DEFAULT_TABLE table=test-table [id=25654ce7dc8f4d42bddfd929ef50d487]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:40.768595 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a049161e3aef4cd39b67be457d116c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:40.773427 17897 tablet_bootstrap.cc:492] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Bootstrap starting.
I20260810 06:25:40.774746 17899 tablet_bootstrap.cc:492] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Bootstrap starting.
I20260810 06:25:40.775784 17897 tablet_bootstrap.cc:654] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:40.775728 17901 tablet_bootstrap.cc:492] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Bootstrap starting.
I20260810 06:25:40.776782 17897 log.cc:824] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:40.776855 17899 tablet_bootstrap.cc:654] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:40.777608 17899 log.cc:824] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:40.777649 17901 tablet_bootstrap.cc:654] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:40.778502 17901 log.cc:824] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:40.779018 17897 tablet_bootstrap.cc:492] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: No bootstrap required, opened a new log
I20260810 06:25:40.779240 17897 ts_tablet_manager.cc:1397] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:25:40.779739 17899 tablet_bootstrap.cc:492] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: No bootstrap required, opened a new log
I20260810 06:25:40.779935 17899 ts_tablet_manager.cc:1397] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:25:40.780288 17901 tablet_bootstrap.cc:492] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: No bootstrap required, opened a new log
I20260810 06:25:40.780505 17901 ts_tablet_manager.cc:1397] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260810 06:25:40.782289 17897 raft_consensus.cc:348] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.782516 17897 raft_consensus.cc:374] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:40.782570 17897 raft_consensus.cc:729] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4640e7d83194665ae03ce5f717bca8b, State: Initialized, Role: FOLLOWER
I20260810 06:25:40.782998 17897 consensus_queue.cc:260] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.783680 17851 heartbeater.cc:503] Master 127.4.73.62:35091 was elected leader, sending a full tablet report...
I20260810 06:25:40.783984 17897 ts_tablet_manager.cc:1428] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:25:40.783860 17899 raft_consensus.cc:348] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.784088 17899 raft_consensus.cc:374] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:40.784148 17899 raft_consensus.cc:729] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28337dc5aec943249975e6eb051466f0, State: Initialized, Role: FOLLOWER
I20260810 06:25:40.784601 17899 consensus_queue.cc:260] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.785542 17899 ts_tablet_manager.cc:1428] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:25:40.785569 17901 raft_consensus.cc:348] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.785811 17901 raft_consensus.cc:374] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:40.785871 17627 heartbeater.cc:503] Master 127.4.73.62:35091 was elected leader, sending a full tablet report...
I20260810 06:25:40.785903 17901 raft_consensus.cc:729] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe9d84a2d9749fb8dd95664f389a48c, State: Initialized, Role: FOLLOWER
I20260810 06:25:40.786285 17901 consensus_queue.cc:260] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.786777 17392 heartbeater.cc:503] Master 127.4.73.62:35091 was elected leader, sending a full tablet report...
I20260810 06:25:40.788206 17901 ts_tablet_manager.cc:1428] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260810 06:25:40.908181 17912 raft_consensus.cc:482] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:40.908459 17912 raft_consensus.cc:504] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.909636 17912 leader_election.cc:288] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827), 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331)
I20260810 06:25:40.913550 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.910060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:58731 (local address 127.4.73.3:44827)
0810 06:25:40.910213 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.910218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.910353 (+ 135us) server_negotiation.cc:407] Connection header received
0810 06:25:40.910512 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.910516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.910573 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.910658 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:40.910934 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.911460 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.912587 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.912985 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.913102 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.913164 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.913342 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.913410 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:25:40.913569 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.909870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:44827 (local address 127.4.73.1:58731)
0810 06:25:40.910228 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.910244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.910392 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.910665 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.910668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.910686 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:40.910811 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.910817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.911578 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.911582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.912461 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.912471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.913105 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.913109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.913205 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.913228 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.913323 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.913400 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":158,"threads_started":1}
I20260810 06:25:40.913851 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.910512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:36979 (local address 127.4.73.2:42331)
0810 06:25:40.910648 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.910653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.910796 (+ 143us) server_negotiation.cc:407] Connection header received
0810 06:25:40.910952 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.910960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.911003 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.911091 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:40.911393 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.911872 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.912888 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.913276 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.913407 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.913483 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.913679 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.913750 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:25:40.913867 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.910354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:42331 (local address 127.4.73.1:36979)
0810 06:25:40.910695 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:40.910709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:40.910835 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:40.911105 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:40.911107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:40.911123 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:40.911257 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.911275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.911994 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.911997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.912755 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:40.912764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.913410 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.913413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:40.913530 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.913552 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:25:40.913651 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 06:25:40.913718 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260810 06:25:40.914316 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a049161e3aef4cd39b67be457d116c71" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4640e7d83194665ae03ce5f717bca8b" is_pre_election: true
I20260810 06:25:40.914381 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a049161e3aef4cd39b67be457d116c71" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0" is_pre_election: true
I20260810 06:25:40.914657 17542 raft_consensus.cc:2454] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 0.
I20260810 06:25:40.914657 17771 raft_consensus.cc:2454] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 0.
I20260810 06:25:40.915149 17187 leader_election.cc:302] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28337dc5aec943249975e6eb051466f0, 6fe9d84a2d9749fb8dd95664f389a48c; no voters:
I20260810 06:25:40.915405 17912 raft_consensus.cc:2781] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:40.915493 17912 raft_consensus.cc:482] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:40.915539 17912 raft_consensus.cc:3037] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:40.916685 17912 raft_consensus.cc:504] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.917024 17912 leader_election.cc:288] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 election: Requested vote from peers a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827), 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331)
I20260810 06:25:40.917373 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a049161e3aef4cd39b67be457d116c71" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4640e7d83194665ae03ce5f717bca8b"
I20260810 06:25:40.917474 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a049161e3aef4cd39b67be457d116c71" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0"
I20260810 06:25:40.917487 17771 raft_consensus.cc:3037] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:40.917632 17542 raft_consensus.cc:3037] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:40.918869 17542 raft_consensus.cc:2454] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 1.
I20260810 06:25:40.918865 17771 raft_consensus.cc:2454] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 1.
I20260810 06:25:40.919188 17187 leader_election.cc:302] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe9d84a2d9749fb8dd95664f389a48c, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:40.919415 17912 raft_consensus.cc:2781] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:40.919646 17912 raft_consensus.cc:686] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 LEADER]: Becoming Leader. State: Replica: 6fe9d84a2d9749fb8dd95664f389a48c, State: Running, Role: LEADER
I20260810 06:25:40.919979 17912 consensus_queue.cc:237] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } }
I20260810 06:25:40.923341 17027 catalog_manager.cc:5495] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1). New cstate: current_term: 1 leader_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } health_report { overall_health: HEALTHY } } }
I20260810 06:25:40.950484 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:40.947070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.4.73.1:43073)
0810 06:25:40.947189 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:40.947194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:40.947215 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:25:40.947417 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:40.947420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:40.947475 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:40.947554 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:40.947832 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.948448 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.949244 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:40.949450 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:40.949525 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:40.949759 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:40.950128 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:40.950209 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:40.950277 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 06:25:40.950322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 06:25:40.952241 17852 tablet.cc:2369] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:40.955986 17771 raft_consensus.cc:1261] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Refusing update from remote peer 6fe9d84a2d9749fb8dd95664f389a48c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:40.955989 17542 raft_consensus.cc:1261] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Refusing update from remote peer 6fe9d84a2d9749fb8dd95664f389a48c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:40.956657 17917 consensus_queue.cc:1035] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:40.956818 17912 consensus_queue.cc:1035] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 06:25:40.962813 17628 tablet.cc:2369] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:40.965806 17931 mvcc.cc:205] Tried to move back new op lower bound from 7316861505349349376 to 7316861505211023360. Current Snapshot: MvccSnapshot[applied={T|T < 7316861505349349376 or (T in {7316861505349349376})}]
I20260810 06:25:40.971869 17027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:40.972026 17027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:40.974258 17027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Sending DeleteTablet for 3 replicas of tablet a049161e3aef4cd39b67be457d116c71
I20260810 06:25:40.975078 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet a049161e3aef4cd39b67be457d116c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:40 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260810 06:25:40.975093 17742 tablet_service.cc:1514] Processing DeleteTablet for tablet a049161e3aef4cd39b67be457d116c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:40 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260810 06:25:40.975534 17942 tablet_replica.cc:330] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: stopping tablet replica
I20260810 06:25:40.975569 17027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:25:40.975718 17269 tablet_service.cc:1514] Processing DeleteTablet for tablet a049161e3aef4cd39b67be457d116c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:40 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60832
I20260810 06:25:40.975816 17942 raft_consensus.cc:2229] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 06:25:40.975921 17027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:40.976104 17942 raft_consensus.cc:2258] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:40.976471 17941 tablet_replica.cc:330] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: stopping tablet replica
I20260810 06:25:40.976850 17941 raft_consensus.cc:2229] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:40.977180 17941 raft_consensus.cc:2258] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:40.977679 17942 ts_tablet_manager.cc:1905] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:40.978681 17941 ts_tablet_manager.cc:1905] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:40.980486 17942 ts_tablet_manager.cc:1918] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:40.980589 17942 log.cc:1196] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/a049161e3aef4cd39b67be457d116c71
I20260810 06:25:40.980942 17942 ts_tablet_manager.cc:1939] T a049161e3aef4cd39b67be457d116c71 P a4640e7d83194665ae03ce5f717bca8b: Deleting consensus metadata
I20260810 06:25:40.981217 17510 tablet_service.cc:1467] Processing CreateTablet for tablet c5da7c49bc234e56990e606d1a05459a (DEFAULT_TABLE table=test-table [id=b50113b26de4440e90c9f2fca18a2b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:40.981411 17742 tablet_service.cc:1467] Processing CreateTablet for tablet c5da7c49bc234e56990e606d1a05459a (DEFAULT_TABLE table=test-table [id=b50113b26de4440e90c9f2fca18a2b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:40.981477 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5da7c49bc234e56990e606d1a05459a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:40.981630 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5da7c49bc234e56990e606d1a05459a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:40.983274 17897 tablet_bootstrap.cc:492] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: Bootstrap starting.
I20260810 06:25:40.983453 17012 catalog_manager.cc:4841] TS a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827): tablet a049161e3aef4cd39b67be457d116c71 (table test-table [id=25654ce7dc8f4d42bddfd929ef50d487]) successfully deleted
I20260810 06:25:40.983811 17269 tablet_service.cc:1467] Processing CreateTablet for tablet c5da7c49bc234e56990e606d1a05459a (DEFAULT_TABLE table=test-table [id=b50113b26de4440e90c9f2fca18a2b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:40.984046 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5da7c49bc234e56990e606d1a05459a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:40.984261 17897 tablet_bootstrap.cc:654] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:40.984890 17943 tablet_replica.cc:330] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: stopping tablet replica
I20260810 06:25:40.985093 17943 raft_consensus.cc:2229] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:40.985353 17943 raft_consensus.cc:2258] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:40.985723 17899 tablet_bootstrap.cc:492] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: Bootstrap starting.
I20260810 06:25:40.986275 17901 tablet_bootstrap.cc:492] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Bootstrap starting.
I20260810 06:25:40.986637 17899 tablet_bootstrap.cc:654] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:40.986668 17943 ts_tablet_manager.cc:1905] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:40.987195 17901 tablet_bootstrap.cc:654] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:40.988719 17899 tablet_bootstrap.cc:492] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: No bootstrap required, opened a new log
I20260810 06:25:40.988806 17899 ts_tablet_manager.cc:1397] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 06:25:40.988992 17941 ts_tablet_manager.cc:1918] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:40.989118 17941 log.cc:1196] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/a049161e3aef4cd39b67be457d116c71
I20260810 06:25:40.989188 17943 ts_tablet_manager.cc:1918] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:40.989286 17943 log.cc:1196] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/a049161e3aef4cd39b67be457d116c71
I20260810 06:25:40.989243 17899 raft_consensus.cc:348] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:40.989341 17899 raft_consensus.cc:374] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:40.989396 17899 raft_consensus.cc:729] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28337dc5aec943249975e6eb051466f0, State: Initialized, Role: FOLLOWER
I20260810 06:25:40.989523 17899 consensus_queue.cc:260] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:40.989585 17943 ts_tablet_manager.cc:1939] T a049161e3aef4cd39b67be457d116c71 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting consensus metadata
I20260810 06:25:40.989768 17899 ts_tablet_manager.cc:1428] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:40.989565 17941 ts_tablet_manager.cc:1939] T a049161e3aef4cd39b67be457d116c71 P 28337dc5aec943249975e6eb051466f0: Deleting consensus metadata
I20260810 06:25:40.991093 17897 tablet_bootstrap.cc:492] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: No bootstrap required, opened a new log
I20260810 06:25:40.991176 17897 ts_tablet_manager.cc:1397] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 06:25:40.991568 17901 tablet_bootstrap.cc:492] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: No bootstrap required, opened a new log
I20260810 06:25:40.991636 17901 ts_tablet_manager.cc:1397] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 06:25:40.991621 17897 raft_consensus.cc:348] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:40.991719 17897 raft_consensus.cc:374] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:40.991750 17897 raft_consensus.cc:729] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4640e7d83194665ae03ce5f717bca8b, State: Initialized, Role: FOLLOWER
I20260810 06:25:40.991858 17897 consensus_queue.cc:260] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:40.991991 17901 raft_consensus.cc:348] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:40.992077 17901 raft_consensus.cc:374] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:40.992102 17897 ts_tablet_manager.cc:1428] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:40.992126 17901 raft_consensus.cc:729] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe9d84a2d9749fb8dd95664f389a48c, State: Initialized, Role: FOLLOWER
I20260810 06:25:40.992229 17901 consensus_queue.cc:260] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:40.992476 17901 ts_tablet_manager.cc:1428] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 06:25:40.992796 17012 catalog_manager.cc:4841] TS 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331): tablet a049161e3aef4cd39b67be457d116c71 (table test-table [id=25654ce7dc8f4d42bddfd929ef50d487]) successfully deleted
I20260810 06:25:40.993377 17012 catalog_manager.cc:4841] TS 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073): tablet a049161e3aef4cd39b67be457d116c71 (table test-table [id=25654ce7dc8f4d42bddfd929ef50d487]) successfully deleted
W20260810 06:25:40.995078 17393 tablet.cc:2369] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:25:41.126940 17912 raft_consensus.cc:482] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:41.127066 17912 raft_consensus.cc:504] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.127396 17912 leader_election.cc:288] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331), a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827)
I20260810 06:25:41.127997 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5da7c49bc234e56990e606d1a05459a" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4640e7d83194665ae03ce5f717bca8b" is_pre_election: true
I20260810 06:25:41.127997 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5da7c49bc234e56990e606d1a05459a" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0" is_pre_election: true
I20260810 06:25:41.128165 17542 raft_consensus.cc:2454] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 0.
I20260810 06:25:41.128165 17771 raft_consensus.cc:2454] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 0.
I20260810 06:25:41.128527 17187 leader_election.cc:302] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe9d84a2d9749fb8dd95664f389a48c, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:41.128705 17912 raft_consensus.cc:2781] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:41.128767 17912 raft_consensus.cc:482] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:41.128826 17912 raft_consensus.cc:3037] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.129758 17912 raft_consensus.cc:504] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.130105 17912 leader_election.cc:288] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 election: Requested vote from peers 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331), a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827)
I20260810 06:25:41.130435 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5da7c49bc234e56990e606d1a05459a" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0"
I20260810 06:25:41.130546 17542 raft_consensus.cc:3037] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.130551 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5da7c49bc234e56990e606d1a05459a" candidate_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4640e7d83194665ae03ce5f717bca8b"
I20260810 06:25:41.130710 17771 raft_consensus.cc:3037] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.131485 17542 raft_consensus.cc:2454] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 1.
I20260810 06:25:41.131685 17771 raft_consensus.cc:2454] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fe9d84a2d9749fb8dd95664f389a48c in term 1.
I20260810 06:25:41.131776 17187 leader_election.cc:302] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28337dc5aec943249975e6eb051466f0, 6fe9d84a2d9749fb8dd95664f389a48c; no voters:
I20260810 06:25:41.131948 17912 raft_consensus.cc:2781] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:41.132097 17912 raft_consensus.cc:686] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 LEADER]: Becoming Leader. State: Replica: 6fe9d84a2d9749fb8dd95664f389a48c, State: Running, Role: LEADER
I20260810 06:25:41.132253 17912 consensus_queue.cc:237] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.133950 17033 catalog_manager.cc:5495] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1). New cstate: current_term: 1 leader_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } health_report { overall_health: UNKNOWN } } }
Aug 10 06:25:41 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343139, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.62@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:41.170125 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.161976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:35091 (local address 127.0.0.1:59790)
0810 06:25:41.162329 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.162354 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.162448 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.162678 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.162680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.163086 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:41.163197 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.163203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.163881 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.163883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.164485 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.164494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.164875 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.164878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.164955 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.165510 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:41.165521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:41.166951 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:41.168765 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:41.168773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:41.168778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:41.169006 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:41.169245 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:41.169247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:41.169248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:41.169323 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:41.169647 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:41.169652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:41.169748 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.169844 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.169976 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":143,"threads_started":1}
I20260810 06:25:41.170151 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.162095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.4.73.62:35091)
0810 06:25:41.162237 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.162242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.162430 (+ 188us) server_negotiation.cc:407] Connection header received
0810 06:25:41.162521 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.162523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.162562 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.162641 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:41.163298 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.163786 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.164583 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.164783 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.164862 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.167085 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:41.167088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:41.167090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:41.167111 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:41.168665 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:41.169091 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:41.169093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:41.169095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:41.169168 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:41.169405 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:41.169407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:41.169408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:41.169564 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:41.169636 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.169901 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.170034 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":62}
I20260810 06:25:41.173704 17033 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:59790:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 06:25:41.174474 17033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:41.179186 17269 tablet_service.cc:1467] Processing CreateTablet for tablet bdc4fe22782a47eba4726d9cc3b6be6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d23f9cdac1244ca7a3090a83d0401726]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:41.179207 17510 tablet_service.cc:1467] Processing CreateTablet for tablet bdc4fe22782a47eba4726d9cc3b6be6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d23f9cdac1244ca7a3090a83d0401726]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:41.179266 17742 tablet_service.cc:1467] Processing CreateTablet for tablet bdc4fe22782a47eba4726d9cc3b6be6a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d23f9cdac1244ca7a3090a83d0401726]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 06:25:41.179621 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc4fe22782a47eba4726d9cc3b6be6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.179621 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc4fe22782a47eba4726d9cc3b6be6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.179629 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdc4fe22782a47eba4726d9cc3b6be6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.182258 17899 tablet_bootstrap.cc:492] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: Bootstrap starting.
I20260810 06:25:41.182509 17897 tablet_bootstrap.cc:492] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: Bootstrap starting.
I20260810 06:25:41.183182 17899 tablet_bootstrap.cc:654] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.183291 17897 tablet_bootstrap.cc:654] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.184401 17901 tablet_bootstrap.cc:492] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: Bootstrap starting.
I20260810 06:25:41.184401 17899 tablet_bootstrap.cc:492] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: No bootstrap required, opened a new log
I20260810 06:25:41.184487 17899 ts_tablet_manager.cc:1397] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:41.184509 17897 tablet_bootstrap.cc:492] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: No bootstrap required, opened a new log
I20260810 06:25:41.184593 17897 ts_tablet_manager.cc:1397] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:41.184926 17899 raft_consensus.cc:348] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.185052 17899 raft_consensus.cc:374] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.185094 17899 raft_consensus.cc:729] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28337dc5aec943249975e6eb051466f0, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.185107 17897 raft_consensus.cc:348] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.185247 17901 tablet_bootstrap.cc:654] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.185250 17897 raft_consensus.cc:374] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.185288 17897 raft_consensus.cc:729] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4640e7d83194665ae03ce5f717bca8b, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.185240 17899 consensus_queue.cc:260] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.185421 17897 consensus_queue.cc:260] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.185729 17897 ts_tablet_manager.cc:1428] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:25:41.185703 17909 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.185885 17909 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:41.185529 17899 ts_tablet_manager.cc:1428] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:41.186313 17910 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.186491 17910 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:41.187494 17901 tablet_bootstrap.cc:492] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: No bootstrap required, opened a new log
I20260810 06:25:41.187590 17901 ts_tablet_manager.cc:1397] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:25:41.187994 17901 raft_consensus.cc:348] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.188143 17901 raft_consensus.cc:374] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.188177 17901 raft_consensus.cc:729] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe9d84a2d9749fb8dd95664f389a48c, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.188288 17901 consensus_queue.cc:260] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.188602 17901 ts_tablet_manager.cc:1428] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:41.189203 17912 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.189335 17912 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:41.365103 17909 raft_consensus.cc:482] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:41.365278 17909 raft_consensus.cc:504] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.366549 17909 leader_election.cc:288] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073), 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331)
I20260810 06:25:41.371423 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.367207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:43875 (local address 127.4.73.2:42331)
0810 06:25:41.367337 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.367344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.367510 (+ 166us) server_negotiation.cc:407] Connection header received
0810 06:25:41.367593 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.367596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.367632 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.367726 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:41.368013 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.368736 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.370183 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.370759 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.370875 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.370932 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.371208 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.371280 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:41.371423 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.366853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:51125 (local address 127.4.73.1:43073)
0810 06:25:41.367033 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.367038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.367076 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:25:41.367249 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.367252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.367303 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.367390 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:41.367709 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.368543 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.370183 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.370762 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.370869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.370935 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.371209 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.371280 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 06:25:41.371469 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.366732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:43073 (local address 127.4.73.3:51125)
0810 06:25:41.366989 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.367002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.367133 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.367419 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.367423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.367451 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:41.367587 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.367594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.368649 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.368652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.370058 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.370069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.370891 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.370895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.371056 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.371110 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.371176 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.371277 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":97,"thread_start_us":68,"threads_started":1}
I20260810 06:25:41.371470 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.367086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:42331 (local address 127.4.73.3:43875)
0810 06:25:41.367411 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.367424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.367509 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.367754 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.367756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.367773 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:41.367898 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.367903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.368834 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.368837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.370064 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.370069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.370891 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.370895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.371062 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.371100 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.371177 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.371277 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":148,"thread_start_us":78,"threads_started":1}
I20260810 06:25:41.372049 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc4fe22782a47eba4726d9cc3b6be6a" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" is_pre_election: true
I20260810 06:25:41.372133 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc4fe22782a47eba4726d9cc3b6be6a" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0" is_pre_election: true
I20260810 06:25:41.372288 17542 raft_consensus.cc:2454] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 0.
I20260810 06:25:41.372344 17296 raft_consensus.cc:2454] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 0.
I20260810 06:25:41.372674 17664 leader_election.cc:302] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28337dc5aec943249975e6eb051466f0, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:41.372933 17909 raft_consensus.cc:2781] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:41.372988 17909 raft_consensus.cc:482] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:41.373013 17909 raft_consensus.cc:3037] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.374122 17909 raft_consensus.cc:504] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.374461 17909 leader_election.cc:288] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 election: Requested vote from peers 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073), 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331)
I20260810 06:25:41.374945 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc4fe22782a47eba4726d9cc3b6be6a" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe9d84a2d9749fb8dd95664f389a48c"
I20260810 06:25:41.374945 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdc4fe22782a47eba4726d9cc3b6be6a" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0"
I20260810 06:25:41.375073 17542 raft_consensus.cc:3037] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.375073 17296 raft_consensus.cc:3037] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.376224 17542 raft_consensus.cc:2454] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 1.
I20260810 06:25:41.376224 17296 raft_consensus.cc:2454] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 1.
I20260810 06:25:41.376552 17664 leader_election.cc:302] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe9d84a2d9749fb8dd95664f389a48c, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:41.376758 17909 raft_consensus.cc:2781] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:41.376981 17909 raft_consensus.cc:686] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [term 1 LEADER]: Becoming Leader. State: Replica: a4640e7d83194665ae03ce5f717bca8b, State: Running, Role: LEADER
I20260810 06:25:41.377254 17909 consensus_queue.cc:237] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.378582 17967 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: TxnStatusTablet state changed. Reason: New leader a4640e7d83194665ae03ce5f717bca8b. Latest consensus state: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.378703 17967 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:41.378988 17970 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 06:25:41.379149 17970 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 06:25:41.379892 17027 catalog_manager.cc:5495] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a4640e7d83194665ae03ce5f717bca8b (127.4.73.3). New cstate: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } health_report { overall_health: HEALTHY } } }
I20260810 06:25:41.420940 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.416631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:44827 (local address 127.0.0.1:55050)
0810 06:25:41.416782 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.416800 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.416900 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.417310 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.417312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.417320 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:41.417419 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.417424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.418487 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.418489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.419162 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.419174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.419795 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.419797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.419864 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.419907 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.420691 (+ 784us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.420705 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.420754 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.420828 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 06:25:41.421005 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.416738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.4.73.3:44827)
0810 06:25:41.416987 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.416991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.417006 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:25:41.417091 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.417095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.417159 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.417298 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:41.417526 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.418383 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.419288 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.419674 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.419791 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.420065 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.420595 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.420698 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.420767 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.420823 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260810 06:25:41.424683 17542 raft_consensus.cc:1261] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Refusing update from remote peer a4640e7d83194665ae03ce5f717bca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.424855 17296 raft_consensus.cc:1261] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Refusing update from remote peer a4640e7d83194665ae03ce5f717bca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.425284 17967 consensus_queue.cc:1035] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.425438 17909 consensus_queue.cc:1035] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.426864 17910 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: TxnStatusTablet state changed. Reason: New leader a4640e7d83194665ae03ce5f717bca8b. Latest consensus state: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.426995 17910 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:41.427486 17912 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: TxnStatusTablet state changed. Reason: New leader a4640e7d83194665ae03ce5f717bca8b. Latest consensus state: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.427603 17912 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:41.430140 17924 mvcc.cc:205] Tried to move back new op lower bound from 7316861507270373376 to 7316861507083976704. Current Snapshot: MvccSnapshot[applied={T|T < 7316861507270373376}]
I20260810 06:25:41.430246 17912 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.430334 17912 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P 6fe9d84a2d9749fb8dd95664f389a48c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:41.430747 17910 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.430855 17910 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P 28337dc5aec943249975e6eb051466f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 06:25:41.432289 17967 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.432417 17967 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:41.432651 17967 tablet_replica.cc:439] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } } }
I20260810 06:25:41.432725 17967 tablet_replica.cc:442] T bdc4fe22782a47eba4726d9cc3b6be6a P a4640e7d83194665ae03ce5f717bca8b: This TxnStatusTablet replica's current role is: LEADER
I20260810 06:25:41.444751 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.441080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:50015 (local address 127.4.73.3:44827)
0810 06:25:41.441204 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.441212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.441416 (+ 204us) server_negotiation.cc:407] Connection header received
0810 06:25:41.441528 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.441530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.441577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.441638 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:41.441909 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.442388 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.443565 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.443788 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.444057 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.444290 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.444432 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.444497 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.444614 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.444659 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:25:41.444842 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.440980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:44827 (local address 127.4.73.1:50015)
0810 06:25:41.441330 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.441345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.441452 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.441670 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.441673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.441681 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:41.441801 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.441807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.442487 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.442491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.443446 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.443457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.444009 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.444014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.444103 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.444155 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.444525 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.444532 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.444608 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.444690 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":103,"threads_started":1}
I20260810 06:25:41.451004 17771 raft_consensus.cc:1261] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Refusing update from remote peer 6fe9d84a2d9749fb8dd95664f389a48c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.451047 17542 raft_consensus.cc:1261] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Refusing update from remote peer 6fe9d84a2d9749fb8dd95664f389a48c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.451403 17917 consensus_queue.cc:1035] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.451538 17912 consensus_queue.cc:1035] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.457358 17930 mvcc.cc:205] Tried to move back new op lower bound from 7316861507379798016 to 7316861506079354880. Current Snapshot: MvccSnapshot[applied={T|T < 7316861507379798016}]
I20260810 06:25:41.480191 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.474763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:47239 (local address 127.4.73.1:43073)
0810 06:25:41.474891 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.474895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.476629 (+ 1734us) server_negotiation.cc:407] Connection header received
0810 06:25:41.476862 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.476866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.476924 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.476979 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:41.477363 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.477881 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.478787 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.479168 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.479265 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.479641 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.479725 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.479776 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.479967 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.480031 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:25:41.480652 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.474721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:43073 (local address 127.4.73.3:47239)
0810 06:25:41.476526 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.476542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.476737 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.477136 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.477140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.477152 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:41.477266 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.477275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.478019 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.478023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.478660 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.478670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.479347 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.479350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.479438 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.479491 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.479844 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.479851 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.480407 (+ 556us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.480491 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1679,"thread_start_us":116,"threads_started":1}
I20260810 06:25:41.507710 17027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:41.507905 17027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:41.510039 17027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Sending DeleteTablet for 3 replicas of tablet c5da7c49bc234e56990e606d1a05459a
I20260810 06:25:41.510596 17269 tablet_service.cc:1514] Processing DeleteTablet for tablet c5da7c49bc234e56990e606d1a05459a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60832
I20260810 06:25:41.510667 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet c5da7c49bc234e56990e606d1a05459a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260810 06:25:41.510901 17742 tablet_service.cc:1514] Processing DeleteTablet for tablet c5da7c49bc234e56990e606d1a05459a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260810 06:25:41.511009 17999 tablet_replica.cc:330] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: stopping tablet replica
I20260810 06:25:41.511153 17999 raft_consensus.cc:2229] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:41.511277 17999 raft_consensus.cc:2258] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.511307 17996 tablet_replica.cc:330] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: stopping tablet replica
I20260810 06:25:41.511447 17996 raft_consensus.cc:2229] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:41.511627 17996 raft_consensus.cc:2258] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.511848 18000 tablet_replica.cc:330] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: stopping tablet replica
I20260810 06:25:41.511951 18000 raft_consensus.cc:2229] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:41.511966 17999 ts_tablet_manager.cc:1905] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.512050 18000 raft_consensus.cc:2258] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.512691 18000 ts_tablet_manager.cc:1905] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.515039 17999 ts_tablet_manager.cc:1918] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:41.515110 17999 log.cc:1196] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/c5da7c49bc234e56990e606d1a05459a
I20260810 06:25:41.515393 17999 ts_tablet_manager.cc:1939] T c5da7c49bc234e56990e606d1a05459a P 28337dc5aec943249975e6eb051466f0: Deleting consensus metadata
I20260810 06:25:41.515666 18000 ts_tablet_manager.cc:1918] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:41.515756 18000 log.cc:1196] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/c5da7c49bc234e56990e606d1a05459a
I20260810 06:25:41.516110 17012 catalog_manager.cc:4841] TS 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331): tablet c5da7c49bc234e56990e606d1a05459a (table test-table [id=b50113b26de4440e90c9f2fca18a2b07]) successfully deleted
I20260810 06:25:41.516727 17996 ts_tablet_manager.cc:1905] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.518953 17996 ts_tablet_manager.cc:1918] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:41.519054 17996 log.cc:1196] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/c5da7c49bc234e56990e606d1a05459a
I20260810 06:25:41.519335 17996 ts_tablet_manager.cc:1939] T c5da7c49bc234e56990e606d1a05459a P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting consensus metadata
I20260810 06:25:41.516358 18000 ts_tablet_manager.cc:1939] T c5da7c49bc234e56990e606d1a05459a P a4640e7d83194665ae03ce5f717bca8b: Deleting consensus metadata
I20260810 06:25:41.520026 17012 catalog_manager.cc:4841] TS 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073): tablet c5da7c49bc234e56990e606d1a05459a (table test-table [id=b50113b26de4440e90c9f2fca18a2b07]) successfully deleted
I20260810 06:25:41.520599 17012 catalog_manager.cc:4841] TS a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827): tablet c5da7c49bc234e56990e606d1a05459a (table test-table [id=b50113b26de4440e90c9f2fca18a2b07]) successfully deleted
Aug 10 06:25:41 dist-test-slave-45dx krb5kdc[16961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 06:25:41 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:41 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:25:41.546329 17122 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:41.534782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.4.73.62:35091)
0810 06:25:41.534930 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.534934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.535076 (+ 142us) server_negotiation.cc:407] Connection header received
0810 06:25:41.535228 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.535231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.535283 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.535371 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:41.536075 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.536945 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.537814 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.538235 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.538354 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.546198 (+ 7844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59800: BlockingRecv error: recv got EOF from 127.0.0.1:59800 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 06:25:41 dist-test-slave-45dx krb5kdc[16961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM
I20260810 06:25:41.556900 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.547961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.4.73.62:35091)
0810 06:25:41.548077 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.548082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.548344 (+ 262us) server_negotiation.cc:407] Connection header received
0810 06:25:41.548450 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.548452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.548531 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.548606 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:41.549311 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.549800 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.550884 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.551114 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.551193 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.553664 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:41.553674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:41.553678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:41.553700 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:41.555128 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:41.555590 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:41.555594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:41.555595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:41.555679 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:41.555975 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:41.555978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:41.555980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:41.556181 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:41.556364 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.556618 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.556734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:25:41.559496 17027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:25:41.559896 17027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:41.564570 17269 tablet_service.cc:1467] Processing CreateTablet for tablet 296afae736424076aaa78443f2f87835 (DEFAULT_TABLE table=test-table [id=bfc00fe25a3248309b1dfe74a833d604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:41.564653 17742 tablet_service.cc:1467] Processing CreateTablet for tablet 296afae736424076aaa78443f2f87835 (DEFAULT_TABLE table=test-table [id=bfc00fe25a3248309b1dfe74a833d604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:41.564653 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 296afae736424076aaa78443f2f87835 (DEFAULT_TABLE table=test-table [id=bfc00fe25a3248309b1dfe74a833d604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:41.564920 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296afae736424076aaa78443f2f87835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.564953 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296afae736424076aaa78443f2f87835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.564954 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 296afae736424076aaa78443f2f87835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.567070 17901 tablet_bootstrap.cc:492] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: Bootstrap starting.
I20260810 06:25:41.567328 17899 tablet_bootstrap.cc:492] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: Bootstrap starting.
I20260810 06:25:41.567971 17901 tablet_bootstrap.cc:654] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.568013 17897 tablet_bootstrap.cc:492] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: Bootstrap starting.
I20260810 06:25:41.568320 17899 tablet_bootstrap.cc:654] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.568912 17897 tablet_bootstrap.cc:654] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.569238 17901 tablet_bootstrap.cc:492] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: No bootstrap required, opened a new log
I20260810 06:25:41.569306 17901 ts_tablet_manager.cc:1397] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:41.569555 17899 tablet_bootstrap.cc:492] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: No bootstrap required, opened a new log
I20260810 06:25:41.569659 17899 ts_tablet_manager.cc:1397] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:41.569752 17901 raft_consensus.cc:348] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.569880 17901 raft_consensus.cc:374] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.569914 17901 raft_consensus.cc:729] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe9d84a2d9749fb8dd95664f389a48c, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.570039 17901 consensus_queue.cc:260] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.570119 17897 tablet_bootstrap.cc:492] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: No bootstrap required, opened a new log
I20260810 06:25:41.570191 17897 ts_tablet_manager.cc:1397] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:41.570307 17901 ts_tablet_manager.cc:1428] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:41.570495 17899 raft_consensus.cc:348] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.570657 17899 raft_consensus.cc:374] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.570684 17899 raft_consensus.cc:729] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28337dc5aec943249975e6eb051466f0, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.570606 17897 raft_consensus.cc:348] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.570725 17897 raft_consensus.cc:374] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.570755 17897 raft_consensus.cc:729] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4640e7d83194665ae03ce5f717bca8b, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.570832 17899 consensus_queue.cc:260] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.570874 17897 consensus_queue.cc:260] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.571163 17897 ts_tablet_manager.cc:1428] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:41.571532 17899 ts_tablet_manager.cc:1428] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:41.657053 17967 raft_consensus.cc:482] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:41.657199 17967 raft_consensus.cc:504] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.657649 17967 leader_election.cc:288] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073), 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331)
I20260810 06:25:41.658133 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296afae736424076aaa78443f2f87835" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" is_pre_election: true
I20260810 06:25:41.658205 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296afae736424076aaa78443f2f87835" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0" is_pre_election: true
I20260810 06:25:41.658272 17296 raft_consensus.cc:2454] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 0.
I20260810 06:25:41.658420 17542 raft_consensus.cc:2454] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 0.
I20260810 06:25:41.658644 17664 leader_election.cc:302] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe9d84a2d9749fb8dd95664f389a48c, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:41.658855 17967 raft_consensus.cc:2781] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:41.658923 17967 raft_consensus.cc:482] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:41.658951 17967 raft_consensus.cc:3037] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.660019 17967 raft_consensus.cc:504] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.660389 17967 leader_election.cc:288] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 election: Requested vote from peers 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073), 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331)
I20260810 06:25:41.660822 17542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296afae736424076aaa78443f2f87835" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28337dc5aec943249975e6eb051466f0"
I20260810 06:25:41.660820 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "296afae736424076aaa78443f2f87835" candidate_uuid: "a4640e7d83194665ae03ce5f717bca8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe9d84a2d9749fb8dd95664f389a48c"
I20260810 06:25:41.660946 17542 raft_consensus.cc:3037] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.660964 17296 raft_consensus.cc:3037] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.662199 17542 raft_consensus.cc:2454] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 1.
I20260810 06:25:41.662199 17296 raft_consensus.cc:2454] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4640e7d83194665ae03ce5f717bca8b in term 1.
I20260810 06:25:41.662562 17664 leader_election.cc:302] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe9d84a2d9749fb8dd95664f389a48c, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:41.662766 17967 raft_consensus.cc:2781] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:41.662895 17967 raft_consensus.cc:686] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 1 LEADER]: Becoming Leader. State: Replica: a4640e7d83194665ae03ce5f717bca8b, State: Running, Role: LEADER
I20260810 06:25:41.663056 17967 consensus_queue.cc:237] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.664732 17027 catalog_manager.cc:5495] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b reported cstate change: term changed from 0 to 1, leader changed from <none> to a4640e7d83194665ae03ce5f717bca8b (127.4.73.3). New cstate: current_term: 1 leader_uuid: "a4640e7d83194665ae03ce5f717bca8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } health_report { overall_health: HEALTHY } } }
I20260810 06:25:41.693773 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.690828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.4.73.3:44827)
0810 06:25:41.690953 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.690957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.690970 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:41.691096 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.691098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.691142 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.691208 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:41.691432 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.691935 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.692831 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.693063 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.693142 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.693391 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.693459 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.693514 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.693615 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.693666 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:25:41.696038 17542 raft_consensus.cc:1261] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Refusing update from remote peer a4640e7d83194665ae03ce5f717bca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.696065 17296 raft_consensus.cc:1261] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Refusing update from remote peer a4640e7d83194665ae03ce5f717bca8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.696533 17967 consensus_queue.cc:1035] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.696700 17981 consensus_queue.cc:1035] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.706018 17027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:41.706167 17027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:41.708065 17027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Sending DeleteTablet for 3 replicas of tablet 296afae736424076aaa78443f2f87835
I20260810 06:25:41.708680 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet 296afae736424076aaa78443f2f87835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260810 06:25:41.708680 17742 tablet_service.cc:1514] Processing DeleteTablet for tablet 296afae736424076aaa78443f2f87835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260810 06:25:41.708868 17999 tablet_replica.cc:330] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: stopping tablet replica
I20260810 06:25:41.708966 17269 tablet_service.cc:1514] Processing DeleteTablet for tablet 296afae736424076aaa78443f2f87835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60832
I20260810 06:25:41.708994 17999 raft_consensus.cc:2229] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:41.709143 17996 tablet_replica.cc:330] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: stopping tablet replica
I20260810 06:25:41.709133 17999 raft_consensus.cc:2258] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.709254 17996 raft_consensus.cc:2229] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:41.709296 18000 tablet_replica.cc:330] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: stopping tablet replica
I20260810 06:25:41.709383 17996 raft_consensus.cc:2258] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.709398 18000 raft_consensus.cc:2229] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:41.709640 18000 raft_consensus.cc:2258] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.710090 17027 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 06:25:41.710358 17996 ts_tablet_manager.cc:1905] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:25:41.710493 17027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:25:41.711498 18000 ts_tablet_manager.cc:1905] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.712864 17996 ts_tablet_manager.cc:1918] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:41.712950 17996 log.cc:1196] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/296afae736424076aaa78443f2f87835
I20260810 06:25:41.712990 17999 ts_tablet_manager.cc:1905] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.713212 17996 ts_tablet_manager.cc:1939] T 296afae736424076aaa78443f2f87835 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting consensus metadata
I20260810 06:25:41.713621 18000 ts_tablet_manager.cc:1918] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:41.713717 18000 log.cc:1196] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/296afae736424076aaa78443f2f87835
I20260810 06:25:41.713840 17012 catalog_manager.cc:4841] TS 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073): tablet 296afae736424076aaa78443f2f87835 (table test-table [id=bfc00fe25a3248309b1dfe74a833d604]) successfully deleted
I20260810 06:25:41.714290 18000 ts_tablet_manager.cc:1939] T 296afae736424076aaa78443f2f87835 P a4640e7d83194665ae03ce5f717bca8b: Deleting consensus metadata
I20260810 06:25:41.714974 17012 catalog_manager.cc:4841] TS a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827): tablet 296afae736424076aaa78443f2f87835 (table test-table [id=bfc00fe25a3248309b1dfe74a833d604]) successfully deleted
I20260810 06:25:41.715466 17999 ts_tablet_manager.cc:1918] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:25:41.715547 17999 log.cc:1196] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/296afae736424076aaa78443f2f87835
I20260810 06:25:41.715814 17999 ts_tablet_manager.cc:1939] T 296afae736424076aaa78443f2f87835 P 28337dc5aec943249975e6eb051466f0: Deleting consensus metadata
I20260810 06:25:41.716506 17012 catalog_manager.cc:4841] TS 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331): tablet 296afae736424076aaa78443f2f87835 (table test-table [id=bfc00fe25a3248309b1dfe74a833d604]) successfully deleted
I20260810 06:25:41.717623 17510 tablet_service.cc:1467] Processing CreateTablet for tablet fcbc9ac81e964d08b2fbf59382e71a60 (DEFAULT_TABLE table=test-table [id=1413d2aad06142a38f4c4d39700448bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:41.717646 17269 tablet_service.cc:1467] Processing CreateTablet for tablet fcbc9ac81e964d08b2fbf59382e71a60 (DEFAULT_TABLE table=test-table [id=1413d2aad06142a38f4c4d39700448bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:41.717878 17269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbc9ac81e964d08b2fbf59382e71a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.717877 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbc9ac81e964d08b2fbf59382e71a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.720028 17899 tablet_bootstrap.cc:492] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: Bootstrap starting.
I20260810 06:25:41.720063 17901 tablet_bootstrap.cc:492] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: Bootstrap starting.
I20260810 06:25:41.720182 17742 tablet_service.cc:1467] Processing CreateTablet for tablet fcbc9ac81e964d08b2fbf59382e71a60 (DEFAULT_TABLE table=test-table [id=1413d2aad06142a38f4c4d39700448bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:25:41.720427 17742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbc9ac81e964d08b2fbf59382e71a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:41.720860 17901 tablet_bootstrap.cc:654] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.720918 17899 tablet_bootstrap.cc:654] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.722090 17901 tablet_bootstrap.cc:492] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: No bootstrap required, opened a new log
I20260810 06:25:41.722168 17901 ts_tablet_manager.cc:1397] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:41.722579 17901 raft_consensus.cc:348] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.722684 17901 raft_consensus.cc:374] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.722723 17901 raft_consensus.cc:729] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe9d84a2d9749fb8dd95664f389a48c, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.722748 17897 tablet_bootstrap.cc:492] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: Bootstrap starting.
I20260810 06:25:41.722872 17901 consensus_queue.cc:260] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.723222 17901 ts_tablet_manager.cc:1428] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:41.723493 17897 tablet_bootstrap.cc:654] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:41.724606 17897 tablet_bootstrap.cc:492] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: No bootstrap required, opened a new log
I20260810 06:25:41.724684 17897 ts_tablet_manager.cc:1397] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 06:25:41.725023 17897 raft_consensus.cc:348] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.725134 17897 raft_consensus.cc:374] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.725167 17897 raft_consensus.cc:729] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4640e7d83194665ae03ce5f717bca8b, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.725282 17897 consensus_queue.cc:260] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.725528 17897 ts_tablet_manager.cc:1428] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:41.726470 17899 tablet_bootstrap.cc:492] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: No bootstrap required, opened a new log
I20260810 06:25:41.726560 17899 ts_tablet_manager.cc:1397] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 06:25:41.727002 17899 raft_consensus.cc:348] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.727115 17899 raft_consensus.cc:374] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:41.727147 17899 raft_consensus.cc:729] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28337dc5aec943249975e6eb051466f0, State: Initialized, Role: FOLLOWER
I20260810 06:25:41.727263 17899 consensus_queue.cc:260] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.727515 17899 ts_tablet_manager.cc:1428] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 06:25:41.772882 17910 raft_consensus.cc:482] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:25:41.773011 17910 raft_consensus.cc:504] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.774143 17910 leader_election.cc:288] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073), a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827)
I20260810 06:25:41.778754 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.774333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:43073 (local address 127.4.73.2:38217)
0810 06:25:41.774628 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.774644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.774797 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.775251 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.775255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.775277 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:41.775425 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.775432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.776517 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.776520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.777406 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.777416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.778297 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.778301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.778405 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.778432 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.778510 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.778577 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260810 06:25:41.778764 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.774456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:38217 (local address 127.4.73.1:43073)
0810 06:25:41.774589 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.774594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.774714 (+ 120us) server_negotiation.cc:407] Connection header received
0810 06:25:41.774970 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.774974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.775028 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.775107 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:41.775559 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.776384 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.777557 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.778170 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.778281 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.778342 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.778545 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.778626 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 06:25:41.779300 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.774775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:44827 (local address 127.4.73.2:46945)
0810 06:25:41.775102 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.775115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.775200 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.775606 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.775609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.775627 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:25:41.775746 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.775752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.776717 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.776720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.777913 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.777925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.778863 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.778867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.778975 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.779005 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.779099 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.779166 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":103,"threads_started":1}
I20260810 06:25:41.779300 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.775099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:46945 (local address 127.4.73.3:44827)
0810 06:25:41.775246 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.775251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.775267 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:25:41.775348 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.775351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.775398 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.775477 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:25:41.775898 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.776601 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.778048 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.778613 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.778716 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.778774 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.779120 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.779195 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 06:25:41.779740 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbc9ac81e964d08b2fbf59382e71a60" candidate_uuid: "28337dc5aec943249975e6eb051466f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4640e7d83194665ae03ce5f717bca8b" is_pre_election: true
I20260810 06:25:41.779901 17771 raft_consensus.cc:2454] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28337dc5aec943249975e6eb051466f0 in term 0.
I20260810 06:25:41.779892 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbc9ac81e964d08b2fbf59382e71a60" candidate_uuid: "28337dc5aec943249975e6eb051466f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" is_pre_election: true
I20260810 06:25:41.780048 17296 raft_consensus.cc:2454] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28337dc5aec943249975e6eb051466f0 in term 0.
I20260810 06:25:41.780251 17435 leader_election.cc:302] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28337dc5aec943249975e6eb051466f0, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:41.780551 17910 raft_consensus.cc:2781] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:25:41.780638 17910 raft_consensus.cc:482] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:25:41.780687 17910 raft_consensus.cc:3037] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.781632 17910 raft_consensus.cc:504] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.782012 17910 leader_election.cc:288] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [CANDIDATE]: Term 1 election: Requested vote from peers 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073), a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827)
I20260810 06:25:41.782416 17771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbc9ac81e964d08b2fbf59382e71a60" candidate_uuid: "28337dc5aec943249975e6eb051466f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4640e7d83194665ae03ce5f717bca8b"
I20260810 06:25:41.782552 17771 raft_consensus.cc:3037] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.783417 17771 raft_consensus.cc:2454] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28337dc5aec943249975e6eb051466f0 in term 1.
I20260810 06:25:41.783684 17296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbc9ac81e964d08b2fbf59382e71a60" candidate_uuid: "28337dc5aec943249975e6eb051466f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe9d84a2d9749fb8dd95664f389a48c"
I20260810 06:25:41.783725 17435 leader_election.cc:302] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28337dc5aec943249975e6eb051466f0, a4640e7d83194665ae03ce5f717bca8b; no voters:
I20260810 06:25:41.783833 17296 raft_consensus.cc:3037] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:41.783912 17910 raft_consensus.cc:2781] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:41.784123 17910 raft_consensus.cc:686] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 1 LEADER]: Becoming Leader. State: Replica: 28337dc5aec943249975e6eb051466f0, State: Running, Role: LEADER
I20260810 06:25:41.784401 17910 consensus_queue.cc:237] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } }
I20260810 06:25:41.785182 17296 raft_consensus.cc:2454] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28337dc5aec943249975e6eb051466f0 in term 1.
I20260810 06:25:41.786886 17033 catalog_manager.cc:5495] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28337dc5aec943249975e6eb051466f0 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "28337dc5aec943249975e6eb051466f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28337dc5aec943249975e6eb051466f0" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 42331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 } health_report { overall_health: UNKNOWN } } }
I20260810 06:25:41.817950 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.813987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.4.73.2:42331)
0810 06:25:41.814119 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.814124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.814139 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:25:41.814312 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.814316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.814384 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.814506 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:41.814703 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.815462 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.816256 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.816652 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.816750 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.817001 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.817579 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.817709 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.817751 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.817818 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:25:41.828557 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.825083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:50075 (local address 127.4.73.3:44827)
0810 06:25:41.825248 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.825252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.825369 (+ 117us) server_negotiation.cc:407] Connection header received
0810 06:25:41.825492 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.825494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.825536 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.825596 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:41.825907 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.826364 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.827508 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.827704 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.827776 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.828141 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.828205 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.828256 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.828403 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.828449 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 06:25:41.828613 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.824932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:44827 (local address 127.4.73.2:50075)
0810 06:25:41.825273 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.825290 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.825400 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.825638 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.825641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.825651 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:41.825774 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.825780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.826476 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.826479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.827391 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.827403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.827858 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.827862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.827961 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.828012 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.828287 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.828294 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.828380 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.828459 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":129,"threads_started":1}
I20260810 06:25:41.834878 17771 raft_consensus.cc:1261] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Refusing update from remote peer 28337dc5aec943249975e6eb051466f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.834955 17296 raft_consensus.cc:1261] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Refusing update from remote peer 28337dc5aec943249975e6eb051466f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:25:41.835528 17910 consensus_queue.cc:1035] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fe9d84a2d9749fb8dd95664f389a48c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 43073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.835738 18048 consensus_queue.cc:1035] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4640e7d83194665ae03ce5f717bca8b" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 44827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:25:41.855438 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.851403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:42331 (local address 127.4.73.3:59405)
0810 06:25:41.851570 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:41.851583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:41.851671 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:41.852160 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:41.852165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:41.852174 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 06:25:41.852335 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.852342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.853169 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.853173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.854040 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:41.854052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.854688 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.854692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:41.854775 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.854820 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.855149 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.855156 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:25:41.855217 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 06:25:41.855300 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 06:25:41.855788 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:41.851439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:59405 (local address 127.4.73.2:42331)
0810 06:25:41.851819 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:41.851823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:41.851836 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:41.851886 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:41.851889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:41.851954 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:41.852029 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:25:41.852471 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.853047 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.854188 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:41.854499 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:41.854604 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:41.854956 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:25:41.855040 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:25:41.855109 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:41.855620 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 06:25:41.855674 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260810 06:25:41.877899 17033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:41.878144 17033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:25:41.880486 17033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdd18b25307c4dba9c9e6c89cfa48144: Sending DeleteTablet for 3 replicas of tablet fcbc9ac81e964d08b2fbf59382e71a60
I20260810 06:25:41.881086 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbc9ac81e964d08b2fbf59382e71a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260810 06:25:41.881244 17999 tablet_replica.cc:330] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: stopping tablet replica
I20260810 06:25:41.881342 17999 raft_consensus.cc:2229] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:25:41.881297 17269 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbc9ac81e964d08b2fbf59382e71a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60832
I20260810 06:25:41.881503 17996 tablet_replica.cc:330] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: stopping tablet replica
I20260810 06:25:41.881587 17999 raft_consensus.cc:2258] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.881613 17996 raft_consensus.cc:2229] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:41.881723 17996 raft_consensus.cc:2258] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.882493 17996 ts_tablet_manager.cc:1905] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.882885 17999 ts_tablet_manager.cc:1905] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.885243 17996 ts_tablet_manager.cc:1918] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:41.885313 17996 log.cc:1196] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/fcbc9ac81e964d08b2fbf59382e71a60
I20260810 06:25:41.885489 17999 ts_tablet_manager.cc:1918] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:41.885564 17996 ts_tablet_manager.cc:1939] T fcbc9ac81e964d08b2fbf59382e71a60 P 6fe9d84a2d9749fb8dd95664f389a48c: Deleting consensus metadata
I20260810 06:25:41.885571 17999 log.cc:1196] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/fcbc9ac81e964d08b2fbf59382e71a60
I20260810 06:25:41.885838 17999 ts_tablet_manager.cc:1939] T fcbc9ac81e964d08b2fbf59382e71a60 P 28337dc5aec943249975e6eb051466f0: Deleting consensus metadata
I20260810 06:25:41.886175 17012 catalog_manager.cc:4841] TS 6fe9d84a2d9749fb8dd95664f389a48c (127.4.73.1:43073): tablet fcbc9ac81e964d08b2fbf59382e71a60 (table test-table [id=1413d2aad06142a38f4c4d39700448bd]) successfully deleted
I20260810 06:25:41.886430 17012 catalog_manager.cc:4841] TS 28337dc5aec943249975e6eb051466f0 (127.4.73.2:42331): tablet fcbc9ac81e964d08b2fbf59382e71a60 (table test-table [id=1413d2aad06142a38f4c4d39700448bd]) successfully deleted
I20260810 06:25:41.886896 17742 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbc9ac81e964d08b2fbf59382e71a60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:25:41 UTC) from {username='oryx', principal='oryx/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260810 06:25:41.887079 18000 tablet_replica.cc:330] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: stopping tablet replica
I20260810 06:25:41.887176 18000 raft_consensus.cc:2229] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:25:41.887275 18000 raft_consensus.cc:2258] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:25:41.887741 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 17155
I20260810 06:25:41.901062 18000 ts_tablet_manager.cc:1905] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:25:41.903100 18000 ts_tablet_manager.cc:1918] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 06:25:41.903204 18000 log.cc:1196] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/fcbc9ac81e964d08b2fbf59382e71a60
I20260810 06:25:41.903450 18000 ts_tablet_manager.cc:1939] T fcbc9ac81e964d08b2fbf59382e71a60 P a4640e7d83194665ae03ce5f717bca8b: Deleting consensus metadata
I20260810 06:25:41.904161 17012 catalog_manager.cc:4841] TS a4640e7d83194665ae03ce5f717bca8b (127.4.73.3:44827): tablet fcbc9ac81e964d08b2fbf59382e71a60 (table test-table [id=1413d2aad06142a38f4c4d39700448bd]) successfully deleted
I20260810 06:25:41.904913 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 17406
I20260810 06:25:41.925168 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 17640
I20260810 06:25:41.944715 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 16982
2026-08-10T06:25:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3604 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:42 dist-test-slave-45dx krb5kdc[18067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:42 dist-test-slave-45dx krb5kdc[18067](info): set up 2 sockets
Aug 10 06:25:42 dist-test-slave-45dx krb5kdc[18067](info): commencing operation
krb5kdc: starting...
W20260810 06:25:43.238895 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:43 dist-test-slave-45dx krb5kdc[18067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:43.361397 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:37849
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:37849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:43.472631 18090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:43.472877 18090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:43.472913 18090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:43.476784 18090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:43.476838 18090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:43.476857 18090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:43.476873 18090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:43.476888 18090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:43.482259 18090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:37849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:37849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:43.483261 18090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:43.484228 18090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:43.493201 18099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:43.493201 18103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:43.493264 18090 server_base.cc:1034] running on GCE node
W20260810 06:25:43.493283 18106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:43.494096 18090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:43.495186 18090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:43.496502 18090 hybrid_clock.cc:648] HybridClock initialized: now 1786343143496467 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T06:25:43Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1556 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:25:43 dist-test-slave-45dx krb5kdc[18115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:25:43 dist-test-slave-45dx krb5kdc[18115](info): set up 2 sockets
Aug 10 06:25:43 dist-test-slave-45dx krb5kdc[18115](info): commencing operation
krb5kdc: starting...
W20260810 06:25:44.770872 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:25:44 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:25:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:25:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:25:44.892515 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43863
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:45.000778 18132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:45.001056 18132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:45.001127 18132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:45.005110 18132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:45.005191 18132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:45.005230 18132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:45.005290 18132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:45.005326 18132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:45.010550 18132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:45.011534 18132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:45.012537 18132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:45.020643 18142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:45.020728 18132 server_base.cc:1034] running on GCE node
W20260810 06:25:45.020637 18145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:45.020637 18140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:45.021404 18132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:45.022504 18132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:45.023691 18132 hybrid_clock.cc:648] HybridClock initialized: now 1786343145023671 us; error 42 us; skew 500 ppm
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:45.026384 18132 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:45.027208 18132 webserver.cc:460] Webserver started at http://127.4.73.62:36009/ using document root <none> and password file <none>
I20260810 06:25:45.027742 18132 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:45.027856 18132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:45.028034 18132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:45.030092 18132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "5ef1c2f19622cce0e19e192b89573ae7"
server_key_iv: "41abcf00658ee1f00541e640d033595e"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.030629 18132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "5ef1c2f19622cce0e19e192b89573ae7"
server_key_iv: "41abcf00658ee1f00541e640d033595e"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.034359 18132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 06:25:45.036620 18157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.037817 18132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:25:45.037966 18132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "5ef1c2f19622cce0e19e192b89573ae7"
server_key_iv: "41abcf00658ee1f00541e640d033595e"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.038077 18132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:45.048985 18132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:45.052052 18132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:45.052246 18132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:45.059078 18132 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43863
I20260810 06:25:45.059090 18243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43863 every 8 connection(s)
I20260810 06:25:45.059963 18132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:45.062577 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:25:45.068725 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 18132
I20260810 06:25:45.068856 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:25:45.069190 4388 external_mini_cluster.cc:1428] Setting key 74dbe8dbbc08e6cacbb43301a37d10cd
I20260810 06:25:45.070268 18245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Bootstrap starting.
I20260810 06:25:45.072990 18245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Neither blocks nor log segments found. Creating new log.
I20260810 06:25:45.073844 18245 log.cc:824] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:45.075876 18245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: No bootstrap required, opened a new log
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:45.079546 18245 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:45.079810 18245 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:25:45.079870 18245 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf9ed465e3d4cc78fc17d25dab9ebd9, State: Initialized, Role: FOLLOWER
I20260810 06:25:45.080295 18245 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:45.080513 18245 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:45.080611 18245 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:45.080703 18245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:25:45.081477 18245 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:45.081869 18245 leader_election.cc:302] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf9ed465e3d4cc78fc17d25dab9ebd9; no voters:
I20260810 06:25:45.082190 18245 leader_election.cc:288] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:25:45.082293 18251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:25:45.082515 18251 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 LEADER]: Becoming Leader. State: Replica: eaf9ed465e3d4cc78fc17d25dab9ebd9, State: Running, Role: LEADER
I20260810 06:25:45.082903 18251 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:45.083381 18245 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:45.084319 18251 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf9ed465e3d4cc78fc17d25dab9ebd9. Latest consensus state: current_term: 1 leader_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } } }
I20260810 06:25:45.084512 18251 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:45.084772 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:45.070602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39048 (local address 127.4.73.62:43863)
0810 06:25:45.071034 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:45.071052 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:45.071091 (+ 39us) server_negotiation.cc:407] Connection header received
0810 06:25:45.071924 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:45.071946 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:45.072386 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:45.072719 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:45.073489 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.074923 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.075765 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.076454 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.076607 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:45.078982 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:45.078989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:45.079004 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:45.079035 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:45.080999 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.081440 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.081446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.081451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.081597 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.081968 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.081970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.081971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.082537 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:45.083322 (+ 785us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:45.083519 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 06:25:45.083808 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":141,"threads_started":1}
I20260810 06:25:45.084766 18252 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } } }
I20260810 06:25:45.084899 18252 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:45.084944 18270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:45.087885 18270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:45.094415 18270 catalog_manager.cc:1347] Generated new cluster ID: 071887a81ddc4fbfafef42fd18f06788
I20260810 06:25:45.094507 18270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:45.107585 18270 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:25:45.108564 18270 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:45.113801 18270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Generated new TSK 0
I20260810 06:25:45.114466 18270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:25:45.145309 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43863
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:45.254508 18295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:45.254724 18295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:45.254762 18295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:45.258628 18295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:45.258677 18295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:45.258760 18295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:45.263710 18295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:45.264649 18295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:45.265563 18295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:45.274746 18302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:45.274794 18304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:45.274863 18295 server_base.cc:1034] running on GCE node
W20260810 06:25:45.274791 18306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:45.275255 18295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:45.275887 18295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:45.277076 18295 hybrid_clock.cc:648] HybridClock initialized: now 1786343145277055 us; error 38 us; skew 500 ppm
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:45.279884 18295 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:25:45.280916 18295 webserver.cc:460] Webserver started at http://127.4.73.1:41813/ using document root <none> and password file <none>
I20260810 06:25:45.281455 18295 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:45.281549 18295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:45.281807 18295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:45.284029 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "8b863450d4094707a8c46fe2bd8d1cf7"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "cde5e99748c612380b1f1b7fb5f93ede"
server_key_iv: "d4c3367772ef96efe79906af727d8340"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.284787 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "8b863450d4094707a8c46fe2bd8d1cf7"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "cde5e99748c612380b1f1b7fb5f93ede"
server_key_iv: "d4c3367772ef96efe79906af727d8340"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.288889 18295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:25:45.291244 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.292464 18295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:45.292623 18295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "8b863450d4094707a8c46fe2bd8d1cf7"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "cde5e99748c612380b1f1b7fb5f93ede"
server_key_iv: "d4c3367772ef96efe79906af727d8340"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.292748 18295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:45.313962 18295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:45.317713 18295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:45.317943 18295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:45.318593 18295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:45.319480 18295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:45.319562 18295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.319648 18295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:45.319698 18295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.329202 18295 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:37399
I20260810 06:25:45.329228 18509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:37399 every 8 connection(s)
I20260810 06:25:45.330149 18295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:45.331528 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 18295
I20260810 06:25:45.331677 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:25:45.331990 4388 external_mini_cluster.cc:1428] Setting key e7cfc3bd62ec3812213531559fd314f4
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:25:45.345191 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:45.332160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:45701 (local address 127.4.73.62:43863)
0810 06:25:45.332338 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:45.332343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:45.333283 (+ 940us) server_negotiation.cc:407] Connection header received
0810 06:25:45.334677 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:45.334680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:45.334737 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:45.334848 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:45.336064 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.336665 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.337853 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.338089 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.338208 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:45.340975 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:45.340979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:45.340981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:45.341006 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:45.342721 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.343366 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.343370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.343371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.343456 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.343802 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.343806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.343808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.344064 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:45.344184 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:45.344886 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 06:25:45.345015 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:25:45.346036 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:45.332665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:45701)
0810 06:25:45.333138 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:45.333190 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:45.334460 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:45.335038 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:45.335045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:45.335411 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:45.335907 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:45.335922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.336796 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.336799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:45.337701 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:45.337711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.338251 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.338255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:45.338435 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:45.339057 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:45.339075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:45.340788 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:45.342855 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:45.342861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:45.342874 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:45.343251 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:45.343551 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:45.343556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:45.343558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:45.343703 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:45.344203 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:45.344210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:45.344606 (+ 396us) client_negotiation.cc:769] Sending connection context
0810 06:25:45.344841 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 06:25:45.345171 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260810 06:25:45.347481 18510 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43863
I20260810 06:25:45.347764 18510 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:45.348272 18510 heartbeater.cc:511] Master 127.4.73.62:43863 requested a full tablet report, sending...
I20260810 06:25:45.351251 18182 ts_manager.cc:194] Registered new tserver with Master: 8b863450d4094707a8c46fe2bd8d1cf7 (127.4.73.1:37399)
I20260810 06:25:45.352391 18182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:45701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:25:45.363325 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43863
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:45.473001 18518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:45.473263 18518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:45.473336 18518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:45.477092 18518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:45.477327 18518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:45.477484 18518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:45.482849 18518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:45.483946 18518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:45.484954 18518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:45.494292 18525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:45.494297 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:45.494364 18518 server_base.cc:1034] running on GCE node
W20260810 06:25:45.494298 18526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:45.494947 18518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:45.495524 18518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:45.496706 18518 hybrid_clock.cc:648] HybridClock initialized: now 1786343145496686 us; error 34 us; skew 500 ppm
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:45.499321 18518 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:25:45.500123 18518 webserver.cc:460] Webserver started at http://127.4.73.2:34777/ using document root <none> and password file <none>
I20260810 06:25:45.500670 18518 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:45.500732 18518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:45.500885 18518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:45.502777 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "e064897b18ac46d897c6c69b559841f9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "858f32faf1ce0a9046adaa3b8446a586"
server_key_iv: "c154ff4b087ab85ab8578e65d242b934"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.503278 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "e064897b18ac46d897c6c69b559841f9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "858f32faf1ce0a9046adaa3b8446a586"
server_key_iv: "c154ff4b087ab85ab8578e65d242b934"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.506714 18518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:25:45.508984 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.510144 18518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:45.510236 18518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "e064897b18ac46d897c6c69b559841f9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "858f32faf1ce0a9046adaa3b8446a586"
server_key_iv: "c154ff4b087ab85ab8578e65d242b934"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.510316 18518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:45.533512 18518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:45.536087 18518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:45.536266 18518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:45.536826 18518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:45.537508 18518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:45.537550 18518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.537585 18518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:45.537604 18518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.546478 18518 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:32771
I20260810 06:25:45.546504 18751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:32771 every 8 connection(s)
I20260810 06:25:45.547291 18518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:45.549433 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 18518
I20260810 06:25:45.549563 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:25:45.549845 4388 external_mini_cluster.cc:1428] Setting key afa518d0dbe420ba6c878011ae6c8fac
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:45.562753 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:45.549017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:56235 (local address 127.4.73.62:43863)
0810 06:25:45.549181 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:45.549186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:45.550095 (+ 909us) server_negotiation.cc:407] Connection header received
0810 06:25:45.551356 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:45.551360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:45.551411 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:45.551508 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:45.552597 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.553470 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.555264 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.555662 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.555795 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:45.559211 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:45.559215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:45.559217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:45.559245 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:45.560692 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.561211 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.561215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.561217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.561300 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.561635 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.561639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.561641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.561851 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:45.561969 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:45.562456 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 06:25:45.562581 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 06:25:45.563664 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:45.549421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:56235)
0810 06:25:45.549939 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:45.549987 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:45.551170 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:45.551655 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:45.551660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:45.551990 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:45.552460 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:45.552473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.553624 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.553639 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:45.555126 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:45.555141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.555872 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.555879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:45.556153 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:45.556955 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:45.556980 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:45.559044 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:45.560835 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:45.560841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:45.560855 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:45.561106 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:45.561392 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:45.561396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:45.561398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:45.561540 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:45.561969 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:45.561977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:45.562215 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 06:25:45.562416 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 06:25:45.562723 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":176,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:25:45.565302 18752 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43863
I20260810 06:25:45.565577 18752 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:45.566210 18752 heartbeater.cc:511] Master 127.4.73.62:43863 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:25:45.567494 18182 ts_manager.cc:194] Registered new tserver with Master: e064897b18ac46d897c6c69b559841f9 (127.4.73.2:32771)
I20260810 06:25:45.568377 18182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:56235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:25:45.583104 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43863
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:45.718933 18760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:45.719198 18760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:45.719281 18760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:45.723248 18760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:45.723330 18760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:45.723415 18760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:45.728504 18760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:45.729453 18760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:45.730414 18760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:45.739867 18772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:45.739867 18768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:45.739938 18769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:45.740175 18760 server_base.cc:1034] running on GCE node
I20260810 06:25:45.740615 18760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:45.741180 18760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:45.742343 18760 hybrid_clock.cc:648] HybridClock initialized: now 1786343145742325 us; error 30 us; skew 500 ppm
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:45.745020 18760 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:25:45.745858 18760 webserver.cc:460] Webserver started at http://127.4.73.3:39855/ using document root <none> and password file <none>
I20260810 06:25:45.746387 18760 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:45.746498 18760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:45.746712 18760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:25:45.748663 18760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "4bb2f0ac8a2642a989915b779925f2e2"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "de7230a717155dc39e000ca800cf911d"
server_key_iv: "89c44e6e364bacc6d2e4ab8043433a99"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.749231 18760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "4bb2f0ac8a2642a989915b779925f2e2"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "de7230a717155dc39e000ca800cf911d"
server_key_iv: "89c44e6e364bacc6d2e4ab8043433a99"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.752593 18760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:25:45.754818 18796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.755925 18760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:45.756062 18760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "4bb2f0ac8a2642a989915b779925f2e2"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "de7230a717155dc39e000ca800cf911d"
server_key_iv: "89c44e6e364bacc6d2e4ab8043433a99"
server_key_version: "encryptionkey@0"
I20260810 06:25:45.756188 18760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:45.775308 18760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:45.778738 18760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:45.778946 18760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:45.779538 18760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:45.780411 18760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:45.780458 18760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.780519 18760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:45.780552 18760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:45.790246 18760 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:44563
I20260810 06:25:45.790295 19024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:44563 every 8 connection(s)
I20260810 06:25:45.791270 18760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:45.797497 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 18760
I20260810 06:25:45.797622 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:25:45.797906 4388 external_mini_cluster.cc:1428] Setting key f4581a8d3d3f77e9b42a26822ae5bb37
Aug 10 06:25:45 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:25:45.805599 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:45.793147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:44783 (local address 127.4.73.62:43863)
0810 06:25:45.793294 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:45.793298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:45.794294 (+ 996us) server_negotiation.cc:407] Connection header received
0810 06:25:45.795627 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:45.795669 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:45.795802 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:45.795888 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:45.797333 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.798146 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.799553 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.799880 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.800016 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:45.802513 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:45.802516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:45.802519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:45.802542 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:45.803775 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.804225 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.804227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.804228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.804355 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:45.804678 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:45.804680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:45.804681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:45.804862 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:45.804938 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:45.805380 (+ 442us) server_negotiation.cc:299] Negotiation successful
0810 06:25:45.805481 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:25:45.806442 19030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:45.793502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:44783)
0810 06:25:45.793963 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:45.794169 (+ 206us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:45.795409 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:45.796111 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:45.796117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:45.796569 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:45.797163 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:45.797178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.798287 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.798292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:45.799445 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:45.799455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:45.800053 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:45.800057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:45.800204 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:45.800892 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:45.800908 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:45.802361 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:25:45.803878 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:45.803883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:45.803895 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:45.804138 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:45.804451 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:25:45.804453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:25:45.804455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:25:45.804589 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:25:45.804952 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:25:45.804957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:25:45.805180 (+ 223us) client_negotiation.cc:769] Sending connection context
0810 06:25:45.805336 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 06:25:45.805576 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260810 06:25:45.807614 19025 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43863
I20260810 06:25:45.807871 19025 heartbeater.cc:464] Registering TS with master...
I20260810 06:25:45.808432 19025 heartbeater.cc:511] Master 127.4.73.62:43863 requested a full tablet report, sending...
I20260810 06:25:45.809505 18182 ts_manager.cc:194] Registered new tserver with Master: 4bb2f0ac8a2642a989915b779925f2e2 (127.4.73.3:44563)
I20260810 06:25:45.810097 18182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:44783
I20260810 06:25:45.812001 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab.
Entry for principal oryx/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab.
WARNING: no policy specified for oryx/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab.
Entry for principal oryx/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab.
WARNING: no policy specified for oryx/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab.
Entry for principal oryx/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab.
I20260810 06:25:45.882934 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 18295
I20260810 06:25:45.899544 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 18518
I20260810 06:25:45.909953 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 18760
I20260810 06:25:45.918094 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 18132
I20260810 06:25:45.932041 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43863
--webserver_interface=127.4.73.62
--webserver_port=36009
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:25:46.040828 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:46.041079 19043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:46.041116 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:46.045046 19043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:25:46.045094 19043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:46.045110 19043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:46.045125 19043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:25:46.045138 19043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:25:46.050092 19043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:46.051045 19043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:46.052031 19043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:46.060474 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:46.060470 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:46.060511 19043 server_base.cc:1034] running on GCE node
W20260810 06:25:46.060489 19058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:46.061296 19043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:46.062453 19043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:46.063627 19043 hybrid_clock.cc:648] HybridClock initialized: now 1786343146063606 us; error 33 us; skew 500 ppm
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343146, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:46.066901 19043 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:25:46.067945 19043 webserver.cc:460] Webserver started at http://127.4.73.62:36009/ using document root <none> and password file <none>
I20260810 06:25:46.068660 19043 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:46.068779 19043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:46.072721 19043 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 06:25:46.074857 19069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.076006 19043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:46.076149 19043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "5ef1c2f19622cce0e19e192b89573ae7"
server_key_iv: "41abcf00658ee1f00541e640d033595e"
server_key_version: "encryptionkey@0"
I20260810 06:25:46.076762 19043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:46.094161 19043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:46.097332 19043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:46.097530 19043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:46.105319 19043 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43863
I20260810 06:25:46.105357 19161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43863 every 8 connection(s)
I20260810 06:25:46.106371 19043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:25:46.108932 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 19043
I20260810 06:25:46.109441 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:37399
--local_ip_for_outbound_sockets=127.4.73.1
--tserver_master_addrs=127.4.73.62:43863
--webserver_port=41813
--webserver_interface=127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:25:46.111217 19163 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:25:46.112977 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Bootstrap starting.
I20260810 06:25:46.129781 19163 log.cc:824] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Log is configured to *not* fsync() on all Append() calls
I20260810 06:25:46.136433 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:25:46.136806 19163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Bootstrap complete.
I20260810 06:25:46.141095 19163 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:46.141518 19163 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf9ed465e3d4cc78fc17d25dab9ebd9, State: Initialized, Role: FOLLOWER
I20260810 06:25:46.142077 19163 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:46.142199 19163 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:25:46.142289 19163 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:25:46.142426 19163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:25:46.144449 19163 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:46.144804 19163 leader_election.cc:302] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaf9ed465e3d4cc78fc17d25dab9ebd9; no voters:
I20260810 06:25:46.145148 19163 leader_election.cc:288] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:25:46.145239 19169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:25:46.145471 19169 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [term 2 LEADER]: Becoming Leader. State: Replica: eaf9ed465e3d4cc78fc17d25dab9ebd9, State: Running, Role: LEADER
I20260810 06:25:46.145767 19169 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } }
I20260810 06:25:46.146413 19163 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:25:46.147159 19170 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } } }
I20260810 06:25:46.147293 19170 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:46.147677 19177 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaf9ed465e3d4cc78fc17d25dab9ebd9. Latest consensus state: current_term: 2 leader_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaf9ed465e3d4cc78fc17d25dab9ebd9" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43863 } } }
I20260810 06:25:46.147774 19177 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9 [sys.catalog]: This master's current role is: LEADER
I20260810 06:25:46.147687 19184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:25:46.151098 19184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:25:46.151779 19184 catalog_manager.cc:1259] Loaded cluster ID: 071887a81ddc4fbfafef42fd18f06788
I20260810 06:25:46.151840 19184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:25:46.154071 19184 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:25:46.154743 19184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eaf9ed465e3d4cc78fc17d25dab9ebd9: Loaded TSK: 0
I20260810 06:25:46.155885 19184 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:25:46.223225 19167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:46.223542 19167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:46.223620 19167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:46.227674 19167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:46.227751 19167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:46.227833 19167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:25:46.232914 19167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.1.keytab
--principal=oryx/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:37399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=41813
--tserver_master_addrs=127.4.73.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:46.233839 19167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:46.234781 19167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:46.244107 19208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:46.244115 19209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:46.244510 19219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:46.245265 19167 server_base.cc:1034] running on GCE node
I20260810 06:25:46.245641 19167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:46.246342 19167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:46.247534 19167 hybrid_clock.cc:648] HybridClock initialized: now 1786343146247511 us; error 41 us; skew 500 ppm
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343146, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:46.250630 19167 init.cc:377] Logged in from keytab as oryx/127.4.73.1@KRBTEST.COM (short username oryx)
I20260810 06:25:46.251614 19167 webserver.cc:460] Webserver started at http://127.4.73.1:41813/ using document root <none> and password file <none>
I20260810 06:25:46.252168 19167 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:46.252274 19167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:46.256047 19167 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:25:46.258260 19234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.259497 19167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:25:46.259588 19167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "8b863450d4094707a8c46fe2bd8d1cf7"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "cde5e99748c612380b1f1b7fb5f93ede"
server_key_iv: "d4c3367772ef96efe79906af727d8340"
server_key_version: "encryptionkey@0"
I20260810 06:25:46.260133 19167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:46.279361 19167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:46.282802 19167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:46.283146 19167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:46.283862 19167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:46.284809 19167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:46.284857 19167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.284919 19167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:46.284950 19167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.295102 19167 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:37399
I20260810 06:25:46.295126 19441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:37399 every 8 connection(s)
I20260810 06:25:46.296077 19167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:25:46.296983 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 19167
I20260810 06:25:46.297451 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:32771
--local_ip_for_outbound_sockets=127.4.73.2
--tserver_master_addrs=127.4.73.62:43863
--webserver_port=34777
--webserver_interface=127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:25:46.310098 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.298098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:46697 (local address 127.4.73.62:43863)
0810 06:25:46.298547 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.298562 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.298961 (+ 399us) server_negotiation.cc:407] Connection header received
0810 06:25:46.301209 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.301226 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.301629 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.302009 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.303050 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.303984 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.305211 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.305648 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.305832 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.309474 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:46697: BlockingRecv error: recv got EOF from 127.4.73.1:46697 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260810 06:25:46.310226 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.298334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:46697)
0810 06:25:46.298809 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.298848 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.300676 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.302065 (+ 1389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.302071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.302415 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.302868 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.302880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.304125 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.304129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.305077 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.305088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.305822 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.305826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.306012 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.306634 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.306649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.309284 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":146,"threads_started":1}
W20260810 06:25:46.310703 19444 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:43863 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.73.62:43863: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:46.315935 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.311139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:53381)
0810 06:25:46.311248 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.311262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.311335 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.311552 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.311555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.311699 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.311781 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.311786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.312551 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.312556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.313084 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.313092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.313545 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.313548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.313644 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.314121 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.314132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.315803 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 06:25:46.316025 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.311227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:53381 (local address 127.4.73.62:43863)
0810 06:25:46.311349 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.311354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.311369 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:25:46.311406 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.311409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.311454 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.311539 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.311892 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.312446 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.313212 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.313431 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.313532 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.315918 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:53381: BlockingRecv error: recv got EOF from 127.4.73.1:53381 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:46.322888 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.316544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:34357)
0810 06:25:46.316629 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.316644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.316713 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.316918 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.316920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.317033 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.317094 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.317098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.318100 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.318136 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.319043 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.319058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.319696 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.319700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.319826 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.320614 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.320634 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.322743 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 06:25:46.323079 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.316609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:34357 (local address 127.4.73.62:43863)
0810 06:25:46.316709 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.316713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.316726 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:46.316782 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.316785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.316820 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.316891 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.317192 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.317936 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.319161 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.319464 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.319571 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.322979 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:34357: BlockingRecv error: recv got EOF from 127.4.73.1:34357 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:25:46.324761 19444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:25:46.429431 19451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:46.429715 19451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:46.429785 19451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:46.433805 19451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:46.433887 19451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:46.433974 19451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:25:46.439450 19451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.2.keytab
--principal=oryx/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:32771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=34777
--tserver_master_addrs=127.4.73.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:46.440614 19451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:46.441591 19451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:46.451357 19458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:46.451462 19459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:46.451443 19451 server_base.cc:1034] running on GCE node
W20260810 06:25:46.451375 19461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:46.451968 19451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:46.452564 19451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:46.453750 19451 hybrid_clock.cc:648] HybridClock initialized: now 1786343146453737 us; error 24 us; skew 500 ppm
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343146, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:46.456422 19451 init.cc:377] Logged in from keytab as oryx/127.4.73.2@KRBTEST.COM (short username oryx)
I20260810 06:25:46.457288 19451 webserver.cc:460] Webserver started at http://127.4.73.2:34777/ using document root <none> and password file <none>
I20260810 06:25:46.457834 19451 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:46.457942 19451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:46.461658 19451 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:46.463630 19471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.464788 19451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:25:46.464921 19451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "e064897b18ac46d897c6c69b559841f9"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "858f32faf1ce0a9046adaa3b8446a586"
server_key_iv: "c154ff4b087ab85ab8578e65d242b934"
server_key_version: "encryptionkey@0"
I20260810 06:25:46.465525 19451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:46.489392 19451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:46.492237 19451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:46.492463 19451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:46.493036 19451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:46.493831 19451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:46.493901 19451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.493966 19451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:46.494009 19451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.502872 19451 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:32771
I20260810 06:25:46.502910 19674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:32771 every 8 connection(s)
I20260810 06:25:46.503793 19451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:25:46.507604 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 19451
I20260810 06:25:46.508095 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:44563
--local_ip_for_outbound_sockets=127.4.73.3
--tserver_master_addrs=127.4.73.62:43863
--webserver_port=39855
--webserver_interface=127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:25:46.515079 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.505552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:56839 (local address 127.4.73.62:43863)
0810 06:25:46.505718 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.505723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.506485 (+ 762us) server_negotiation.cc:407] Connection header received
0810 06:25:46.507680 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.507685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.507751 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.507841 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.509109 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.509947 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.511057 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.511310 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.511427 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.514944 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:56839: BlockingRecv error: recv got EOF from 127.4.73.2:56839 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 06:25:46.515637 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.505917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:56839)
0810 06:25:46.506334 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.506374 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.507471 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.508003 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.508008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.508500 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.508949 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.508963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.510081 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.510086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.510923 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.510933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.511468 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.511472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.511654 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.512221 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.512236 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.514827 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":117,"threads_started":1}
W20260810 06:25:46.516122 19675 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:43863 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.73.62:43863: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:46.522428 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.516648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:57107)
0810 06:25:46.516848 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.516865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.516960 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.517222 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.517228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.517437 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.517595 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.517605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.518328 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.518333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.519380 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.519397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.519838 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.519845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.519976 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.520491 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.520504 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.522306 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110}
W20260810 06:25:46.522636 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.516731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:57107 (local address 127.4.73.62:43863)
0810 06:25:46.516861 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.516864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.516911 (+ 47us) server_negotiation.cc:407] Connection header received
0810 06:25:46.517068 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.517070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.517104 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.517168 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.517733 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.518184 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.519527 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.519724 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.519794 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.522421 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:57107: BlockingRecv error: recv got EOF from 127.4.73.2:57107 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:46.528235 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.523056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:57149)
0810 06:25:46.523169 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.523186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.523262 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.523486 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.523489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.523640 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.523708 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.523713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.524478 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.524482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.524996 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.525005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.525477 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.525481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.525565 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.526081 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.526093 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.528081 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 06:25:46.528369 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.523140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:57149 (local address 127.4.73.62:43863)
0810 06:25:46.523250 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.523254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.523269 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:25:46.523340 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.523344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.523387 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.523470 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.523827 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.524362 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.525135 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.525362 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.525470 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.528213 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:57149: BlockingRecv error: recv got EOF from 127.4.73.2:57149 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 06:25:46.528548 19675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:25:46.623096 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:25:46.623377 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:25:46.623452 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:25:46.627345 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:25:46.627432 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:25:46.627517 19679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:25:46.632514 19679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/krb5kdc/oryx_127.4.73.3.keytab
--principal=oryx/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:44563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=39855
--tserver_master_addrs=127.4.73.62:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:25:46.633500 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:25:46.634469 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:25:46.644109 19690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:46.644138 19695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:25:46.644140 19691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:25:46.644476 19679 server_base.cc:1034] running on GCE node
I20260810 06:25:46.644915 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:25:46.645511 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:25:46.646720 19679 hybrid_clock.cc:648] HybridClock initialized: now 1786343146646707 us; error 23 us; skew 500 ppm
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343146, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:25:46.655439 19679 init.cc:377] Logged in from keytab as oryx/127.4.73.3@KRBTEST.COM (short username oryx)
I20260810 06:25:46.656263 19679 webserver.cc:460] Webserver started at http://127.4.73.3:39855/ using document root <none> and password file <none>
I20260810 06:25:46.656893 19679 fs_manager.cc:359] Metadata directory not provided
I20260810 06:25:46.656968 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:25:46.660784 19679 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 06:25:46.662678 19706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.663941 19679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:25:46.664070 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "4bb2f0ac8a2642a989915b779925f2e2"
format_stamp: "Formatted at 2026-08-10 06:25:45 on dist-test-slave-45dx"
server_key: "de7230a717155dc39e000ca800cf911d"
server_key_iv: "89c44e6e364bacc6d2e4ab8043433a99"
server_key_version: "encryptionkey@0"
I20260810 06:25:46.664716 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:25:46.685062 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:25:46.687871 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:25:46.688090 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:25:46.688750 19679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:25:46.689659 19679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:25:46.689709 19679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.689777 19679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:25:46.689812 19679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:25:46.698724 19679 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:44563
I20260810 06:25:46.698766 19905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:44563 every 8 connection(s)
I20260810 06:25:46.699692 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:25:46.705366 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 19679
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 06:25:46.711017 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.701382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:51543 (local address 127.4.73.62:43863)
0810 06:25:46.701507 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.701513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.702323 (+ 810us) server_negotiation.cc:407] Connection header received
0810 06:25:46.703635 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.703639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.703700 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.703802 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.704977 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.705731 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.707243 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.707452 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.707599 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.710917 (+ 3318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:51543: BlockingRecv error: recv got EOF from 127.4.73.3:51543 (error 108)
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":46}
I20260810 06:25:46.711699 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.701672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:51543)
0810 06:25:46.702168 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.702210 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.703428 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.703978 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.703985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.704369 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.704817 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.704831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.705883 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.705886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.707114 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.707130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.707604 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.707608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.707744 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.708376 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.708402 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.710817 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":175,"threads_started":1}
W20260810 06:25:46.712145 19906 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:43863 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.73.62:43863: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
I20260810 06:25:46.715113 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.706332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.4.73.62:43863)
0810 06:25:46.706534 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.706538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.706549 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:25:46.706578 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.706580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.706612 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.706686 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.707551 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.708110 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.709074 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.709340 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.709445 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.710804 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:25:46.710811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:25:46.710827 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:25:46.710857 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:25:46.713095 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:46.713533 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:46.713537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:46.713541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:46.713653 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:25:46.713932 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:25:46.713934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:25:46.713935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:25:46.714324 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:25:46.714521 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 06:25:46.714716 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 06:25:46.714925 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:46.717837 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.712603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:40179)
0810 06:25:46.712757 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.712770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.712846 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.713080 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.713083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.713241 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.713336 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.713341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.714147 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.714150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.714939 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.714950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.715398 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.715401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.715468 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.715863 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.715873 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.717718 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260810 06:25:46.717947 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.712683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:40179 (local address 127.4.73.62:43863)
0810 06:25:46.712789 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.712792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.712806 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:46.712943 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.712945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.712975 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.713037 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.713458 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.714047 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.715059 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.715297 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.715385 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.717831 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:40179: BlockingRecv error: recv got EOF from 127.4.73.3:40179 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:46 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:46.723250 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:46.718518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:55079)
0810 06:25:46.718645 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:46.718660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:46.718737 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:46.718990 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:46.718993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:46.719142 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:46.719208 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.719213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.719903 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.719907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.720491 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:46.720499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.720955 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.720958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:46.721043 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.721491 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:46.721502 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:46.723120 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 06:25:46.723313 19448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:46.718552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:55079 (local address 127.4.73.62:43863)
0810 06:25:46.718714 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:46.718718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:46.718730 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:25:46.718842 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:46.718845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:46.718881 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:46.718975 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:46.719318 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.719797 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.720614 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:46.720834 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:46.720952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:46.723217 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:55079: BlockingRecv error: recv got EOF from 127.4.73.3:55079 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260810 06:25:46.723582 19906 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 06:25:47 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:47 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:47.331156 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:47.325390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:51451)
0810 06:25:47.325695 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:47.325709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:47.325798 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:47.326029 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:47.326032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:47.326217 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:47.326308 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:47.326313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.327192 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.327195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:47.328040 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:47.328051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.328577 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.328580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:47.328644 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:47.329136 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:47.329148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:47.331035 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
W20260810 06:25:47.331312 19913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:47.325505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:51451 (local address 127.4.73.62:43863)
0810 06:25:47.325796 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:47.325800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:47.325813 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:47.325872 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:47.325874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:47.325914 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:47.325991 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:47.326414 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.327066 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.328181 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.328469 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.328555 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:47.331170 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:51451: BlockingRecv error: recv got EOF from 127.4.73.1:51451 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
Aug 10 06:25:47 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:47 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:47.534974 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:47.529189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:58535)
0810 06:25:47.529487 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:47.529502 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:47.529654 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:47.529941 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:47.529944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:47.530152 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:47.530245 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:47.530251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.531042 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.531047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:47.531627 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:47.531636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.532117 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.532120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:47.532206 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:47.532727 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:47.532738 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:47.534841 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":88,"threads_started":1}
W20260810 06:25:47.535207 19913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:47.529291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:58535 (local address 127.4.73.62:43863)
0810 06:25:47.529451 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:47.529455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:47.529607 (+ 152us) server_negotiation.cc:407] Connection header received
0810 06:25:47.529766 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:47.529769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:47.529822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:47.529913 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:47.530369 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.530925 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.531764 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.532000 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.532110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:47.535046 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:58535: BlockingRecv error: recv got EOF from 127.4.73.2:58535 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 06:25:47 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:47 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:47.729481 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:47.724216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:59471)
0810 06:25:47.724541 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:47.724555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:47.724658 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:47.724889 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:47.724891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:47.725075 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:47.725203 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:47.725208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.725910 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.725913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:47.726505 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:47.726512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.726904 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.726907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:47.726976 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:47.727520 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:47.727532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:47.729361 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":130,"threads_started":1}
W20260810 06:25:47.729650 19913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:47.724258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:59471 (local address 127.4.73.62:43863)
0810 06:25:47.724409 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:47.724414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:47.724633 (+ 219us) server_negotiation.cc:407] Connection header received
0810 06:25:47.724740 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:47.724742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:47.724788 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:47.724866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:47.725300 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.725812 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.726603 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:47.726811 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:47.726895 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:47.729495 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:59471: BlockingRecv error: recv got EOF from 127.4.73.3:59471 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 06:25:48 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:48 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:48.337635 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:48.331939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:60389)
0810 06:25:48.332288 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:48.332328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:48.332494 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:48.332742 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:48.332746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:48.332979 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:48.333084 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:48.333091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.333828 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.333831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:48.334448 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:48.334457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.334981 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.334985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:48.335055 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:48.335547 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:48.335558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:48.337499 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":72,"threads_started":1}
W20260810 06:25:48.337800 19920 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:48.332096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:60389 (local address 127.4.73.62:43863)
0810 06:25:48.332520 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:48.332523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:48.332536 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:48.332573 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:48.332575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:48.332617 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:48.332698 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:48.333193 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.333718 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.334554 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.334844 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.334953 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:48.337630 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:60389: BlockingRecv error: recv got EOF from 127.4.73.1:60389 (error 108)
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":112,"threads_started":1}
Aug 10 06:25:48 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:48 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:48.541069 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:48.535814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:54355)
0810 06:25:48.536099 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:48.536113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:48.536215 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:48.536524 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:48.536526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:48.536710 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:48.536806 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:48.536810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.537502 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.537504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:48.538028 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:48.538034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.538407 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.538410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:48.538494 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:48.538987 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:48.539000 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:48.540942 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
W20260810 06:25:48.541144 19920 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:48.535956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:54355 (local address 127.4.73.62:43863)
0810 06:25:48.536117 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:48.536120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:48.536163 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:25:48.536295 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:48.536297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:48.536412 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:48.536480 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:48.536895 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.537413 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.538115 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.538306 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.538372 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:48.541059 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:54355: BlockingRecv error: recv got EOF from 127.4.73.2:54355 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 06:25:48 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:48 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:48.735570 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:48.730227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:44969)
0810 06:25:48.730544 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:48.730558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:48.730655 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:48.730887 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:48.730889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:48.731071 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:48.731170 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:48.731175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.731924 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.731927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:48.732565 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:48.732575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.732978 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.732980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:48.733042 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:48.733550 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:48.733562 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:48.735433 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":139,"threads_started":1}
W20260810 06:25:48.735767 19920 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:48.730363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:44969 (local address 127.4.73.62:43863)
0810 06:25:48.730523 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:48.730527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:48.730623 (+ 96us) server_negotiation.cc:407] Connection header received
0810 06:25:48.730720 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:48.730722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:48.730768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:48.730838 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:48.731311 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.731829 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.732670 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:48.732876 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:48.732958 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:48.735636 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:44969: BlockingRecv error: recv got EOF from 127.4.73.3:44969 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 10 06:25:49 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:49 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:49.344142 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:49.338480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:39583)
0810 06:25:49.338779 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:49.338793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:49.338888 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:49.339155 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:49.339157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:49.339348 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:49.339454 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:49.339460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.340170 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.340172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:49.340816 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:49.340826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.341417 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.341420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:49.341520 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:49.342088 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:49.342102 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:49.343993 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
W20260810 06:25:49.344308 19931 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:49.338617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:39583 (local address 127.4.73.62:43863)
0810 06:25:49.338843 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:49.338846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:49.338860 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:49.338984 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:49.338986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:49.339028 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:49.339106 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:49.339555 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.340071 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.340938 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.341265 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.341387 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:49.344143 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:39583: BlockingRecv error: recv got EOF from 127.4.73.1:39583 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
Aug 10 06:25:49 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:49 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:49.547175 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:49.541927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:59081)
0810 06:25:49.542218 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:49.542232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:49.542330 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:49.542560 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:49.542562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:49.542739 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:49.542836 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:49.542841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.543549 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.543552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:49.544074 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:49.544080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.544490 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.544492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:49.544552 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:49.545110 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:49.545122 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:49.546949 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
W20260810 06:25:49.547254 19931 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:49.541962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:59081 (local address 127.4.73.62:43863)
0810 06:25:49.542121 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:49.542126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:49.542305 (+ 179us) server_negotiation.cc:407] Connection header received
0810 06:25:49.542410 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:49.542413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:49.542457 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:49.542525 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:49.542944 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.543459 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.544168 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.544400 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.544474 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:49.547115 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:59081: BlockingRecv error: recv got EOF from 127.4.73.2:59081 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 06:25:49 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:49 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:49.742141 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:49.736408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:44625)
0810 06:25:49.736694 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:49.736706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:49.736849 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:49.737151 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:49.737154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:49.737369 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:49.737467 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:49.737473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.738233 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.738238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:49.738862 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:49.738871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.739331 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.739335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:49.739425 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:49.739942 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:49.739953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:49.741953 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
W20260810 06:25:49.742249 19931 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:49.736486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:44625 (local address 127.4.73.62:43863)
0810 06:25:49.736626 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:49.736631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:49.736778 (+ 147us) server_negotiation.cc:407] Connection header received
0810 06:25:49.736951 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:49.736954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:49.737027 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:49.737119 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:49.737586 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.738122 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.738987 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:49.739213 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:49.739326 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:49.742116 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:44625: BlockingRecv error: recv got EOF from 127.4.73.3:44625 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 06:25:50 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:50 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:50.350947 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:50.344910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:44449)
0810 06:25:50.345188 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:50.345201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:50.345302 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:50.345549 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:50.345551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:50.345743 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:50.345864 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:50.345869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.346684 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.346687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:50.347374 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:50.347388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.347805 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.347808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:50.347870 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:50.348495 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:50.348513 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:50.350765 (+ 2252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
W20260810 06:25:50.351004 19936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:50.344992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:44449 (local address 127.4.73.62:43863)
0810 06:25:50.345246 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:50.345250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:50.345262 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:25:50.345393 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:50.345396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:50.345437 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:50.345516 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:50.345972 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.346587 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.347487 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.347705 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.347783 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:50.350886 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:44449: BlockingRecv error: recv got EOF from 127.4.73.1:44449 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
Aug 10 06:25:50 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:50 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:50.553380 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:50.548101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:45477)
0810 06:25:50.548412 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:50.548426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:50.548524 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:50.548784 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:50.548786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:50.548977 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:50.549078 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:50.549083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.549790 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.549792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:50.550365 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:50.550373 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.550809 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.550811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:50.550881 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:50.551383 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:50.551394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:50.553270 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
W20260810 06:25:50.553556 19936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:50.548174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:45477 (local address 127.4.73.62:43863)
0810 06:25:50.548389 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:50.548393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:50.548491 (+ 98us) server_negotiation.cc:407] Connection header received
0810 06:25:50.548602 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:50.548622 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:50.548665 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:50.548732 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:50.549176 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.549697 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.550491 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.550701 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.550796 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:50.553420 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:45477: BlockingRecv error: recv got EOF from 127.4.73.2:45477 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 06:25:50 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:50 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:50.748438 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:50.742947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:53971)
0810 06:25:50.743203 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:50.743217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:50.743316 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:50.743558 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:50.743560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:50.743742 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:50.743839 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:50.743843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.744639 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.744642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:50.745196 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:50.745203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.745612 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.745614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:50.745674 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:50.746219 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:50.746239 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:50.748239 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
W20260810 06:25:50.748502 19936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:50.743051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:53971 (local address 127.4.73.62:43863)
0810 06:25:50.743217 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:50.743221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:50.743288 (+ 67us) server_negotiation.cc:407] Connection header received
0810 06:25:50.743398 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:50.743401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:50.743448 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:50.743518 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:50.743948 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.744536 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.745299 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:50.745513 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:50.745594 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:50.748385 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:53971: BlockingRecv error: recv got EOF from 127.4.73.3:53971 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 06:25:51 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:51 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:51.357414 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:51.351809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:55609)
0810 06:25:51.352119 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:51.352135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:51.352242 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:51.352539 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:51.352542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:51.352728 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:51.352828 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:51.352833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.353661 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.353664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:51.354270 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:51.354279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.354683 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.354685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:51.354745 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:51.355259 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:51.355273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:51.357231 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
W20260810 06:25:51.357461 19944 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:51.351861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:55609 (local address 127.4.73.62:43863)
0810 06:25:51.352156 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:51.352159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:51.352201 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:25:51.352374 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:51.352376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:51.352418 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:51.352504 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:51.352947 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.353558 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.354377 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.354588 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.354660 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:51.357346 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:55609: BlockingRecv error: recv got EOF from 127.4.73.1:55609 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":123,"threads_started":1}
Aug 10 06:25:51 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:51 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:51.559888 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:51.554122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:54103)
0810 06:25:51.554419 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:51.554433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:51.554520 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:51.554742 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:51.554744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:51.554929 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:51.555026 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:51.555032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.555731 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.555734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:51.556340 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:51.556357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.556798 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.556801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:51.556865 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:51.557496 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:51.557510 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:51.559739 (+ 2229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
W20260810 06:25:51.560005 19944 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:51.554386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:54103 (local address 127.4.73.62:43863)
0810 06:25:51.554529 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:51.554532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:51.554566 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:25:51.554601 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:51.554603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:51.554644 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:51.554702 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:51.555127 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.555639 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.556454 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.556695 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.556767 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:51.559873 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:54103: BlockingRecv error: recv got EOF from 127.4.73.2:54103 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 06:25:51 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:51 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:51.759665 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:51.749270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:40001)
0810 06:25:51.749577 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:51.749591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:51.749684 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:51.749925 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:51.749927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:51.750108 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:51.750206 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:51.750211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.750919 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.750921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:51.751498 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:51.751505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.751907 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.751910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:51.751975 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:51.757591 (+ 5616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:51.757603 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:51.759534 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":132,"threads_started":1}
W20260810 06:25:51.759786 19944 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:51.749369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:40001 (local address 127.4.73.62:43863)
0810 06:25:51.749511 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:51.749515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:51.749658 (+ 143us) server_negotiation.cc:407] Connection header received
0810 06:25:51.749763 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:51.749765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:51.749808 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:51.749879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:51.750305 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.750824 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.751597 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:51.751808 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:51.751879 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:51.759666 (+ 7787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:40001: BlockingRecv error: recv got EOF from 127.4.73.3:40001 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 06:25:52 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:52 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:52.364141 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:52.358388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:33017)
0810 06:25:52.358722 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:52.358737 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:52.358876 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:52.359195 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:52.359198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:52.359466 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:52.359592 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:52.359598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.360413 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.360418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:52.360996 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:52.361005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.361454 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.361456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:52.361518 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:52.362037 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:52.362049 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:52.364018 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":94,"threads_started":1}
W20260810 06:25:52.364437 19953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:52.358489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:33017 (local address 127.4.73.62:43863)
0810 06:25:52.358814 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:52.358817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:52.358830 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:52.358997 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:52.359000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:52.359055 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:52.359167 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:52.359712 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.360250 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.361116 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.361352 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.361436 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:52.364158 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:33017: BlockingRecv error: recv got EOF from 127.4.73.1:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
Aug 10 06:25:52 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:52 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:52.565910 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:52.560719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:53859)
0810 06:25:52.560989 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:52.561002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:52.561100 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:52.561307 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:52.561309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:52.561485 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:52.561579 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:52.561584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.562373 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.562375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:52.562939 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:52.562946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.563367 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.563369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:52.563429 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:52.563892 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:52.563902 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:52.565748 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
W20260810 06:25:52.566067 19953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:52.560759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:53859 (local address 127.4.73.62:43863)
0810 06:25:52.560922 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:52.560926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:52.561069 (+ 143us) server_negotiation.cc:407] Connection header received
0810 06:25:52.561165 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:52.561167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:52.561208 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:52.561271 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:52.561708 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.562274 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.563060 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.563272 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.563356 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:52.565926 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:53859: BlockingRecv error: recv got EOF from 127.4.73.2:53859 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 06:25:52 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:52 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:52.766647 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:52.760631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:57791)
0810 06:25:52.760958 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:52.760994 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:52.761125 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:52.761461 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:52.761466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:52.761693 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:52.761851 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:52.761886 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.762686 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.762690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:52.763340 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:52.763351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.763858 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.763862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:52.763936 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:52.764545 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:52.764557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:52.766473 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":127,"threads_started":1}
W20260810 06:25:52.766817 19953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:52.760720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:57791 (local address 127.4.73.62:43863)
0810 06:25:52.760868 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:52.760873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:52.761053 (+ 180us) server_negotiation.cc:407] Connection header received
0810 06:25:52.761248 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:52.761251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:52.761313 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:52.761408 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:52.762012 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.762567 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.763489 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:52.763711 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:52.763802 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:52.766687 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:57791: BlockingRecv error: recv got EOF from 127.4.73.3:57791 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 06:25:53 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:53 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:53.370615 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:53.364896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:51263)
0810 06:25:53.365307 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:53.365321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:53.365424 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:53.365667 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:53.365669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:53.365892 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:53.365996 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:53.366002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.366796 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.366799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:53.367341 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:53.367350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.367777 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.367780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:53.367842 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:53.368368 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:53.368386 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:53.370453 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":89,"threads_started":1}
W20260810 06:25:53.370872 19959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:53.364995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:51263 (local address 127.4.73.62:43863)
0810 06:25:53.365251 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:53.365255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:53.365392 (+ 137us) server_negotiation.cc:407] Connection header received
0810 06:25:53.365503 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:53.365505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:53.365549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:53.365626 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:53.366092 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.366661 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.367451 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.367666 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.367735 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:53.370717 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:51263: BlockingRecv error: recv got EOF from 127.4.73.1:51263 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
Aug 10 06:25:53 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:53 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:53.572738 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:53.566756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:41249)
0810 06:25:53.567265 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:53.567278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:53.567420 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:53.567712 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:53.567715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:53.567980 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:53.568128 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:53.568139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.568956 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.568958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:53.569635 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:53.569645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.570068 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.570070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:53.570153 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:53.570689 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:53.570719 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:53.572624 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":94,"threads_started":1}
W20260810 06:25:53.572878 19959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:53.566909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:41249 (local address 127.4.73.62:43863)
0810 06:25:53.567061 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:53.567065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:53.567374 (+ 309us) server_negotiation.cc:407] Connection header received
0810 06:25:53.567533 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:53.567535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:53.567580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:53.567669 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:53.568268 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.568831 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.569739 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.569960 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.570041 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:53.572761 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:41249: BlockingRecv error: recv got EOF from 127.4.73.2:41249 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 06:25:53 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:53 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:53.773619 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:53.767508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:59295)
0810 06:25:53.767872 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:53.767886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:53.767984 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:53.768237 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:53.768239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:53.768500 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:53.768659 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:53.768666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.769484 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.769488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:53.770189 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:53.770199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.770640 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.770642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:53.770705 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:53.771274 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:53.771286 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:53.773446 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":131,"threads_started":1}
W20260810 06:25:53.773681 19959 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:53.767579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:59295 (local address 127.4.73.62:43863)
0810 06:25:53.767716 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:53.767720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:53.767960 (+ 240us) server_negotiation.cc:407] Connection header received
0810 06:25:53.768072 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:53.768075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:53.768123 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:53.768198 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:53.768806 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.769348 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.770304 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:53.770539 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:53.770620 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:53.773570 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:59295: BlockingRecv error: recv got EOF from 127.4.73.3:59295 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 06:25:54 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:54 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:54.376694 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:54.371474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:46593)
0810 06:25:54.371763 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:54.371780 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:54.371887 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:54.372104 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:54.372106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:54.372287 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:54.372406 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:54.372411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.373112 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.373115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:54.373625 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:54.373633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.374059 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.374062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:54.374129 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:54.374619 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:54.374631 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:54.376580 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":68,"threads_started":1}
W20260810 06:25:54.376893 19966 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:54.371647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:46593 (local address 127.4.73.62:43863)
0810 06:25:54.371893 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:54.371896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:54.371909 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:54.371944 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:54.371946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:54.371985 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:54.372059 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:54.372508 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.373012 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.373723 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.373956 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.374030 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:54.376710 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:46593: BlockingRecv error: recv got EOF from 127.4.73.1:46593 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
Aug 10 06:25:54 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:54 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:54.579710 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:54.573682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:35849)
0810 06:25:54.574060 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:54.574073 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:54.574218 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:54.574541 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:54.574545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:54.574828 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:54.574931 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:54.574937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.575839 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.575843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:54.576515 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:54.576525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.576976 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.576980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:54.577067 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:54.577572 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:54.577584 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:54.579536 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":127,"threads_started":1}
W20260810 06:25:54.579773 19966 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:54.573721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:35849 (local address 127.4.73.62:43863)
0810 06:25:54.573895 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:54.573900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:54.574159 (+ 259us) server_negotiation.cc:407] Connection header received
0810 06:25:54.574330 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:54.574333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:54.574391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:54.574505 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:54.575054 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.575720 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.576636 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.576860 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.576972 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:54.579663 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:35849: BlockingRecv error: recv got EOF from 127.4.73.2:35849 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 06:25:54 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:54 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:54.780714 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:54.774649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:37687)
0810 06:25:54.774996 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:54.775009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:54.775142 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:54.775429 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:54.775432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:54.775648 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:54.775747 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:54.775752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.776575 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.776581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:54.777151 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:54.777159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.777632 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.777635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:54.777732 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:54.778247 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:54.778258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:54.780598 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":147,"threads_started":1}
W20260810 06:25:54.780972 19966 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:54.774714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:37687 (local address 127.4.73.62:43863)
0810 06:25:54.774892 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:54.774897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:54.775089 (+ 192us) server_negotiation.cc:407] Connection header received
0810 06:25:54.775253 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:54.775256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:54.775310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:54.775402 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:54.775880 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.776467 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.777275 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:54.777505 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:54.777614 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:54.780726 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:37687: BlockingRecv error: recv got EOF from 127.4.73.3:37687 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 06:25:55 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:55 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:55.383162 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:55.377516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:57575)
0810 06:25:55.377796 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:55.377809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:55.377900 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:55.378161 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:55.378163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:55.378363 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:55.378463 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:55.378468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.379259 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.379262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:55.379832 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:55.379841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.380385 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.380390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:55.380473 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:55.381060 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:55.381073 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:55.383037 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
W20260810 06:25:55.383314 19973 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:55.377591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:57575 (local address 127.4.73.62:43863)
0810 06:25:55.377835 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:55.377844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:55.377857 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:25:55.377996 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:55.377998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:55.378045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:55.378123 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:55.378563 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.379117 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.379941 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.380162 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.380243 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:55.383174 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:57575: BlockingRecv error: recv got EOF from 127.4.73.1:57575 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":62,"threads_started":1}
Aug 10 06:25:55 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:55 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:55.585816 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:55.580477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:42101)
0810 06:25:55.580754 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:55.580769 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:55.580871 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:55.581168 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:55.581170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:55.581364 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:55.581466 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:55.581471 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.582240 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.582244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:55.582820 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:55.582829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.583229 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.583231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:55.583296 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:55.583819 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:55.583842 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:55.585680 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
W20260810 06:25:55.585952 19973 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:55.580656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:42101 (local address 127.4.73.62:43863)
0810 06:25:55.580789 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:55.580793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:55.580873 (+ 80us) server_negotiation.cc:407] Connection header received
0810 06:25:55.581000 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:55.581002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:55.581047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:55.581122 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:55.581568 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.582137 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.582926 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.583133 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.583208 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:55.585835 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:42101: BlockingRecv error: recv got EOF from 127.4.73.2:42101 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 06:25:55 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:55 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:55.787207 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:55.781512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:36841)
0810 06:25:55.781793 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:55.781808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:55.781911 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:55.782138 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:55.782140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:55.782340 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:55.782438 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:55.782444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.783429 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.783432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:55.784055 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:55.784066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.784569 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.784572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:55.784636 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:55.785117 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:55.785128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:55.787091 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260810 06:25:55.787477 19973 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:55.781569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:36841 (local address 127.4.73.62:43863)
0810 06:25:55.781703 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:55.781707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:55.781882 (+ 175us) server_negotiation.cc:407] Connection header received
0810 06:25:55.781979 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:55.781981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:55.782027 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:55.782097 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:55.782568 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.783285 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.784171 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:55.784465 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:55.784549 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:55.787230 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:36841: BlockingRecv error: recv got EOF from 127.4.73.3:36841 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 06:25:56 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:56 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:56.392020 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:56.384103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:42633)
0810 06:25:56.384438 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:56.384452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:56.384543 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:56.384773 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:56.384775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:56.384953 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:56.385078 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:56.385088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.386098 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.386101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:56.387026 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:56.387038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.387779 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.387786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:56.387897 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:56.388486 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:56.388505 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:56.391839 (+ 3334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":87,"threads_started":1}
W20260810 06:25:56.392292 19984 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:56.384152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:42633 (local address 127.4.73.62:43863)
0810 06:25:56.384444 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:56.384447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:56.384519 (+ 72us) server_negotiation.cc:407] Connection header received
0810 06:25:56.384619 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:56.384621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:56.384663 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:56.384740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:56.385384 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.385991 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.387212 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.387623 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.387770 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:56.391981 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:42633: BlockingRecv error: recv got EOF from 127.4.73.1:42633 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":60,"threads_started":1}
Aug 10 06:25:56 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:56 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:56.592465 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:56.586840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:52997)
0810 06:25:56.587132 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:56.587145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:56.587239 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:56.587491 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:56.587493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:56.587729 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:56.587874 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:56.587882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.588950 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.588954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:56.589531 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:56.589539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.589943 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.589945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:56.590009 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:56.590522 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:56.590533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:56.592295 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":122,"threads_started":1}
W20260810 06:25:56.592595 19984 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:56.586945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:52997 (local address 127.4.73.62:43863)
0810 06:25:56.587084 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:56.587088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:56.587214 (+ 126us) server_negotiation.cc:407] Connection header received
0810 06:25:56.587321 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:56.587323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:56.587366 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:56.587439 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:56.588012 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.588830 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.589629 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.589844 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.589924 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:56.592483 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:52997: BlockingRecv error: recv got EOF from 127.4.73.2:52997 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 06:25:56 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:56 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:56.796983 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:56.788133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:41769)
0810 06:25:56.788458 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:56.788472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:56.788559 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:56.788798 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:56.788800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:56.788981 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:56.789081 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:56.789086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.789816 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.789818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:56.790381 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:56.790389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.790785 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.790787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:56.790847 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:56.794868 (+ 4021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:56.794880 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:56.796805 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":82,"threads_started":1}
W20260810 06:25:56.797020 19984 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:56.788390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:41769 (local address 127.4.73.62:43863)
0810 06:25:56.788532 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:56.788540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:56.788554 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:56.788644 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:56.788646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:56.788690 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:56.788757 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:56.789178 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.789714 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.790481 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:56.790690 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:56.790764 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:56.796921 (+ 6157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:41769: BlockingRecv error: recv got EOF from 127.4.73.3:41769 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 06:25:57 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:57 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:57.399030 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:57.392874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:47165)
0810 06:25:57.393152 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:57.393168 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:57.393279 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:57.393581 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:57.393584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:57.393842 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:57.393989 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:57.393997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.395112 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.395118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:57.395820 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:57.395830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.396327 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.396331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:57.396419 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:57.396971 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:57.396991 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:57.398840 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
W20260810 06:25:57.399075 19997 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:57.392952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:47165 (local address 127.4.73.62:43863)
0810 06:25:57.393210 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:57.393215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:57.393232 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:25:57.393392 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:57.393396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:57.393448 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:57.393533 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:57.394130 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.394995 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.395943 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.396182 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.396319 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:57.398959 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:47165: BlockingRecv error: recv got EOF from 127.4.73.1:47165 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
Aug 10 06:25:57 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:57 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:57.599108 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:57.593344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:40123)
0810 06:25:57.593674 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:57.593694 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:57.593811 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:57.594115 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:57.594118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:57.594399 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:57.594500 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:57.594506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.595284 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.595289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:57.595856 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:57.595863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.596361 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.596365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:57.596454 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:57.596988 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:57.596999 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:57.598923 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
W20260810 06:25:57.599145 19997 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:57.593406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:40123 (local address 127.4.73.62:43863)
0810 06:25:57.593669 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:57.593673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:57.593843 (+ 170us) server_negotiation.cc:407] Connection header received
0810 06:25:57.593915 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:57.593918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:57.593978 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:57.594085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:57.594623 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.595165 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.595989 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.596206 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.596310 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:57.599045 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:40123: BlockingRecv error: recv got EOF from 127.4.73.2:40123 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 06:25:57 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:57 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:57.803404 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:57.797740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:60871)
0810 06:25:57.798073 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:57.798086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:57.798187 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:57.798428 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:57.798430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:57.798624 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:57.798726 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:57.798731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.799466 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.799469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:57.800022 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:57.800030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.800545 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.800548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:57.800609 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:57.801216 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:57.801228 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:57.803284 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":154,"threads_started":1}
W20260810 06:25:57.803624 19997 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:57.797837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:60871 (local address 127.4.73.62:43863)
0810 06:25:57.797984 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:57.797987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:57.798186 (+ 199us) server_negotiation.cc:407] Connection header received
0810 06:25:57.798256 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:57.798259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:57.798308 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:57.798382 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:57.798833 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.799361 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.800137 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:57.800444 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:57.800528 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:57.803424 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:60871: BlockingRecv error: recv got EOF from 127.4.73.3:60871 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 06:25:58 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:58 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:58.405596 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:58.399951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:49779)
0810 06:25:58.400222 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:58.400235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:58.400388 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:58.400696 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:58.400699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:58.400897 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:58.401000 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:58.401005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.401819 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.401822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:58.402392 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:58.402401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.402858 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.402861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:58.402957 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:58.403487 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:58.403499 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:58.405478 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
W20260810 06:25:58.405789 20004 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:58.400014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:49779 (local address 127.4.73.62:43863)
0810 06:25:58.400243 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:58.400253 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:58.400285 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:25:58.400497 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:58.400499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:58.400544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:58.400621 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:58.401111 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.401665 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.402507 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.402718 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.402792 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:58.405620 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:49779: BlockingRecv error: recv got EOF from 127.4.73.1:49779 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":64,"threads_started":1}
Aug 10 06:25:58 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:58 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:58.606066 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:58.599980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:42955)
0810 06:25:58.600325 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:58.600340 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:58.600488 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:58.600772 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:58.600775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:58.600997 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:58.601091 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:58.601097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.601905 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.601915 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:58.602522 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:58.602531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.603171 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.603175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:58.603269 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:58.603837 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:58.603856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:58.605952 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
W20260810 06:25:58.606225 20004 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:58.600052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:42955 (local address 127.4.73.62:43863)
0810 06:25:58.600230 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:58.600235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:58.600427 (+ 192us) server_negotiation.cc:407] Connection header received
0810 06:25:58.600607 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:58.600610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:58.600664 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:58.600750 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:58.601211 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.601731 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.602644 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.602977 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.603126 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:58.606098 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:42955: BlockingRecv error: recv got EOF from 127.4.73.2:42955 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 06:25:58 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:58 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:58.810178 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:58.804187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:48271)
0810 06:25:58.804530 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:58.804546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:58.804662 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:58.804920 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:58.804922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:58.805130 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:58.805254 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:58.805260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.806023 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.806026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:58.806676 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:58.806686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.807123 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.807125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:58.807196 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:58.807704 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:58.807716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:58.809988 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":146,"threads_started":1}
W20260810 06:25:58.810258 20004 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:58.804268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:48271 (local address 127.4.73.62:43863)
0810 06:25:58.804427 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:58.804432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:58.804625 (+ 193us) server_negotiation.cc:407] Connection header received
0810 06:25:58.804745 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:58.804747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:58.804799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:58.804868 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:58.805367 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.805898 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.806793 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:58.807012 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:58.807092 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:58.810136 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:48271: BlockingRecv error: recv got EOF from 127.4.73.3:48271 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 06:25:59 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:59 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:59.412078 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:59.406420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:34115)
0810 06:25:59.406688 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:59.406704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:59.406824 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:59.407088 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:59.407090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:59.407268 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:59.407399 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:59.407404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.408171 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.408177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:59.408933 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:59.408944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.409521 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.409524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:59.409590 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:59.410085 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:59.410097 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:59.411955 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
W20260810 06:25:59.412236 20011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:59.406514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:34115 (local address 127.4.73.62:43863)
0810 06:25:59.406765 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:59.406770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:59.406784 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:25:59.406910 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:59.406912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:59.406956 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:59.407050 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:59.407503 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.408049 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.409027 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.409380 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.409482 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:59.412105 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:34115: BlockingRecv error: recv got EOF from 127.4.73.1:34115 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
Aug 10 06:25:59 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:59 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:59.612830 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:59.606852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:37669)
0810 06:25:59.607200 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:59.607217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:59.607345 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:59.607642 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:59.607644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:59.607872 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:59.607969 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:59.607975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.608852 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.608857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:59.609438 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:59.609447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.609927 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.609930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:59.610019 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:59.610609 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:59.610623 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:59.612703 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
W20260810 06:25:59.613029 20011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:59.607051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:37669 (local address 127.4.73.62:43863)
0810 06:25:59.607201 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:59.607207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:59.607285 (+ 78us) server_negotiation.cc:407] Connection header received
0810 06:25:59.607469 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:59.607472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:59.607528 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:59.607614 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:59.608098 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.608729 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.609564 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.609807 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.609921 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:59.612855 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:37669: BlockingRecv error: recv got EOF from 127.4.73.2:37669 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 06:25:59 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:25:59 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:25:59.816278 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:25:59.811107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:53635)
0810 06:25:59.811420 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 06:25:59.811435 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:25:59.811541 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:25:59.811763 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:25:59.811765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:25:59.811954 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:25:59.812051 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:59.812056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.812781 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.812784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:59.813368 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:25:59.813376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.813793 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.813796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:25:59.813873 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:59.814373 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:25:59.814383 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:25:59.816168 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":133,"threads_started":1}
W20260810 06:25:59.816478 20011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:25:59.811179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:53635 (local address 127.4.73.62:43863)
0810 06:25:59.811331 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 06:25:59.811336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:25:59.811518 (+ 182us) server_negotiation.cc:407] Connection header received
0810 06:25:59.811608 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:25:59.811611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:25:59.811656 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:25:59.811723 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:25:59.812151 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.812683 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.813476 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:25:59.813696 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:25:59.813774 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:25:59.816327 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:53635: BlockingRecv error: recv got EOF from 127.4.73.3:53635 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 06:26:00 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:26:00 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:26:00.418572 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:00.412886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:42707)
0810 06:26:00.413185 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:00.413198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:00.413294 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:00.413638 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:00.413641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:00.413937 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:00.414055 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:00.414061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.415075 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.415078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:00.415611 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:00.415619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.416006 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.416009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:00.416072 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:00.416606 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:00.416617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:00.418449 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
W20260810 06:26:00.418725 20015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:00.412964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:42707 (local address 127.4.73.62:43863)
0810 06:26:00.413284 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:00.413288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:00.413303 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:26:00.413422 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:00.413425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:00.413482 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:00.413591 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:00.414173 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.414965 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.415705 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.415913 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.415987 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:00.418582 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:42707: BlockingRecv error: recv got EOF from 127.4.73.1:42707 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
Aug 10 06:26:00 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:26:00 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:26:00.619061 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:00.613601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:38483)
0810 06:26:00.613896 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:00.613919 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:00.614025 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:00.614255 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:00.614257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:00.614474 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:00.614578 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:00.614583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.615313 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.615316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:00.615869 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:00.615876 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.616343 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.616348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:00.616446 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:00.617064 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:00.617076 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:00.618933 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":117,"threads_started":1}
W20260810 06:26:00.619190 20015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:00.613695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:38483 (local address 127.4.73.62:43863)
0810 06:26:00.613855 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:00.613859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:00.613992 (+ 133us) server_negotiation.cc:407] Connection header received
0810 06:26:00.614092 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:00.614094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:00.614138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:00.614209 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:00.614680 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.615201 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.615971 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.616181 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.616257 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:00.619069 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:38483: BlockingRecv error: recv got EOF from 127.4.73.2:38483 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 06:26:00 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:26:00 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.3@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:26:00.822983 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:00.817192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.3:44267)
0810 06:26:00.817564 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:00.817580 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:00.817682 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:00.817932 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:00.817934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:00.818133 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:00.818236 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:00.818241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.818980 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.818983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:00.819565 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:00.819573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.819986 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.819988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:00.820051 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:00.820675 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:00.820687 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:00.822788 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":185,"threads_started":1}
W20260810 06:26:00.823062 20015 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:00.817284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:44267 (local address 127.4.73.62:43863)
0810 06:26:00.817437 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:00.817441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:00.817651 (+ 210us) server_negotiation.cc:407] Connection header received
0810 06:26:00.817763 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:00.817766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:00.817814 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:00.817893 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:00.818348 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.818873 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.819673 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:00.819887 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:00.819964 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:00.822930 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.3:44267: BlockingRecv error: recv got EOF from 127.4.73.3:44267 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 06:26:01 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:26:01 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.1@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:26:01.424906 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:01.419554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.1:33365)
0810 06:26:01.419826 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:01.419840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:01.419923 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:01.420164 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:01.420166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:01.420415 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:01.420518 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:01.420524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.421250 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.421253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:01.421842 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:01.421850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.422290 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.422292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:01.422358 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:01.422858 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:01.422870 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:01.424775 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
W20260810 06:26:01.425034 20024 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:01.419624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:33365 (local address 127.4.73.62:43863)
0810 06:26:01.419903 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:01.419907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:01.419920 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:26:01.420004 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:01.420007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:01.420047 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:01.420131 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:01.420628 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.421155 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.421955 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.422180 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.422261 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:01.424905 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.1:33365: BlockingRecv error: recv got EOF from 127.4.73.1:33365 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":126,"threads_started":1}
Aug 10 06:26:01 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 06:26:01 dist-test-slave-45dx krb5kdc[18115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.73.2@KRBTEST.COM for oryx/127.4.73.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 06:26:01.626190 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:01.620073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43863 (local address 127.4.73.2:51047)
0810 06:26:01.620379 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:01.620394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:01.620482 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:01.620743 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:01.620745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:01.620955 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:01.621058 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:01.621063 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.621895 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.621898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:01.622467 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:01.622475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.622972 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.622974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:01.623036 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:01.623567 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:01.623579 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:01.625970 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.73.62:43863: Server oryx/127.4.73.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
W20260810 06:26:01.626269 20024 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:01.620226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:51047 (local address 127.4.73.62:43863)
0810 06:26:01.620418 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:01.620422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:01.620439 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:26:01.620575 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:01.620577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:01.620625 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:01.620708 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:01.621160 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.621739 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.622591 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:01.622864 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:01.622955 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:01.626133 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.73.2:51047: BlockingRecv error: recv got EOF from 127.4.73.2:51047 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 06:26:01.705857 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 19167
I20260810 06:26:01.721976 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 19451
I20260810 06:26:01.736407 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 19679
I20260810 06:26:01.776930 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 19043
2026-08-10T06:26:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18274 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T06:26:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:26:01Z Disabled control of system clock
I20260810 06:26:01.816625 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38689
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:38689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 06:26:01.929416 20033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:01.929682 20033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:01.929718 20033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:01.933755 20033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:26:01.933811 20033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:01.933827 20033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:01.933844 20033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:26:01.933859 20033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:26:01.938510 20033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:38689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:01.939492 20033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:01.940828 20033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:26:01.949538 20033 server_base.cc:1034] running on GCE node
W20260810 06:26:01.949641 20044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:01.949607 20042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:01.949782 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:01.950472 20033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:01.951505 20033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:01.952696 20033 hybrid_clock.cc:648] HybridClock initialized: now 1786343161952692 us; error 40 us; skew 500 ppm
I20260810 06:26:01.959043 20033 webserver.cc:460] Webserver started at http://127.4.73.62:35385/ using document root <none> and password file <none>
I20260810 06:26:01.959643 20033 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:01.959765 20033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:01.960016 20033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:01.962173 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "5585da6b69e1436da6db6b3fc1d085ed"
format_stamp: "Formatted at 2026-08-10 06:26:01 on dist-test-slave-45dx"
server_key: "bdd9b8528953e4a9d96acb20a8694e4b"
server_key_iv: "69b1dc7eeb4a377478826d62ece076b3"
server_key_version: "encryptionkey@0"
I20260810 06:26:01.962751 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "5585da6b69e1436da6db6b3fc1d085ed"
format_stamp: "Formatted at 2026-08-10 06:26:01 on dist-test-slave-45dx"
server_key: "bdd9b8528953e4a9d96acb20a8694e4b"
server_key_iv: "69b1dc7eeb4a377478826d62ece076b3"
server_key_version: "encryptionkey@0"
I20260810 06:26:01.966507 20033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:26:01.969044 20051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:01.970623 20033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:26:01.970726 20033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "5585da6b69e1436da6db6b3fc1d085ed"
format_stamp: "Formatted at 2026-08-10 06:26:01 on dist-test-slave-45dx"
server_key: "bdd9b8528953e4a9d96acb20a8694e4b"
server_key_iv: "69b1dc7eeb4a377478826d62ece076b3"
server_key_version: "encryptionkey@0"
I20260810 06:26:01.970821 20033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:01.986604 20033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:01.987166 20033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:01.987305 20033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:01.994096 20033 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:38689
I20260810 06:26:01.994164 20145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:38689 every 8 connection(s)
I20260810 06:26:01.995036 20033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:26:01.998260 20147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:02.002688 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 20033
I20260810 06:26:02.002781 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:26:02.003094 4388 external_mini_cluster.cc:1428] Setting key 97f39278a379ce83f340e10a82436461
I20260810 06:26:02.004063 20147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed: Bootstrap starting.
I20260810 06:26:02.006507 20147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:02.007261 20147 log.cc:824] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:02.009186 20147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed: No bootstrap required, opened a new log
I20260810 06:26:02.012730 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.004620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.4.73.62:38689)
0810 06:26:02.005283 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:02.005302 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:02.005340 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:26:02.005989 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:02.006006 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:02.006420 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:02.006815 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:02.007279 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.008532 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.009456 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.009958 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.010130 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:02.011023 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:02.011028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:02.011039 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:02.011409 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:02.011466 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:02.011486 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:02.011599 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:02.011713 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 06:26:02.011968 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":128,"threads_started":1}
I20260810 06:26:02.012676 20147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5585da6b69e1436da6db6b3fc1d085ed" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38689 } }
I20260810 06:26:02.012976 20147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:02.013053 20147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5585da6b69e1436da6db6b3fc1d085ed, State: Initialized, Role: FOLLOWER
I20260810 06:26:02.013505 20147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5585da6b69e1436da6db6b3fc1d085ed" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38689 } }
I20260810 06:26:02.013662 20147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:26:02.013731 20147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:26:02.013854 20147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:02.014747 20147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5585da6b69e1436da6db6b3fc1d085ed" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38689 } }
I20260810 06:26:02.015120 20147 leader_election.cc:302] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5585da6b69e1436da6db6b3fc1d085ed; no voters:
I20260810 06:26:02.015363 20147 leader_election.cc:288] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:26:02.015491 20153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:26:02.015727 20153 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [term 1 LEADER]: Becoming Leader. State: Replica: 5585da6b69e1436da6db6b3fc1d085ed, State: Running, Role: LEADER
I20260810 06:26:02.016059 20153 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5585da6b69e1436da6db6b3fc1d085ed" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38689 } }
I20260810 06:26:02.016587 20147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:26:02.017814 20154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5585da6b69e1436da6db6b3fc1d085ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5585da6b69e1436da6db6b3fc1d085ed" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38689 } } }
I20260810 06:26:02.017814 20159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5585da6b69e1436da6db6b3fc1d085ed. Latest consensus state: current_term: 1 leader_uuid: "5585da6b69e1436da6db6b3fc1d085ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5585da6b69e1436da6db6b3fc1d085ed" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38689 } } }
I20260810 06:26:02.017997 20154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [sys.catalog]: This master's current role is: LEADER
I20260810 06:26:02.018028 20159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed [sys.catalog]: This master's current role is: LEADER
I20260810 06:26:02.019116 20171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:26:02.021698 20171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:26:02.026765 20171 catalog_manager.cc:1347] Generated new cluster ID: 08726967db674b138ee13ec80012bc98
I20260810 06:26:02.026827 20171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:26:02.038347 20171 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:26:02.039515 20171 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:26:02.064220 20171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5585da6b69e1436da6db6b3fc1d085ed: Generated new TSK 0
I20260810 06:26:02.065008 20171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:26:02.073187 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38689
--builtin_ntp_servers=127.4.73.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 06:26:02.227226 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:02.227481 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:02.227536 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:02.231467 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:02.231518 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:02.231581 20190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:26:02.236030 20190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:02.237062 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:02.238379 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:02.248512 20209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.248518 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.248518 20200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:02.248607 20190 server_base.cc:1034] running on GCE node
I20260810 06:26:02.249241 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:02.249836 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:02.250998 20190 hybrid_clock.cc:648] HybridClock initialized: now 1786343162250980 us; error 29 us; skew 500 ppm
I20260810 06:26:02.252971 20190 webserver.cc:460] Webserver started at http://127.4.73.1:37535/ using document root <none> and password file <none>
I20260810 06:26:02.253573 20190 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:02.253639 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:02.253867 20190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:02.255959 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "7d2a6f1a1802455a87a236a85461dac9"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "47e658e02948d37919b401c11bfd800d"
server_key_iv: "38bef24b144b95c0e1d26c61fb89d1b4"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.256544 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "7d2a6f1a1802455a87a236a85461dac9"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "47e658e02948d37919b401c11bfd800d"
server_key_iv: "38bef24b144b95c0e1d26c61fb89d1b4"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.260085 20190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:26:02.263269 20222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.265092 20190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:26:02.265237 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "7d2a6f1a1802455a87a236a85461dac9"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "47e658e02948d37919b401c11bfd800d"
server_key_iv: "38bef24b144b95c0e1d26c61fb89d1b4"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.265394 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:02.277928 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:02.278508 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:02.278654 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:02.279158 20190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:02.280068 20190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:02.280117 20190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.280158 20190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:02.280174 20190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.289063 20190 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:46609
I20260810 06:26:02.289093 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:46609 every 8 connection(s)
I20260810 06:26:02.289959 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:26:02.291312 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 20190
I20260810 06:26:02.291431 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:26:02.291769 4388 external_mini_cluster.cc:1428] Setting key 6dcc72ca0362f953339e2beb31d7aa27
I20260810 06:26:02.294021 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38689
--builtin_ntp_servers=127.4.73.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 06:26:02.300670 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.291817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:34615 (local address 127.4.73.62:38689)
0810 06:26:02.291958 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:02.291962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:02.293257 (+ 1295us) server_negotiation.cc:407] Connection header received
0810 06:26:02.294831 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:02.294835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:02.294889 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:02.294970 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:02.296097 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.296736 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.298486 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.298700 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.298778 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:02.300159 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:02.300162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:02.300164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:02.300177 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:02.300196 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:02.300204 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:02.300262 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:02.300507 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 06:26:02.300558 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:26:02.301960 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.292366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38689 (local address 127.4.73.1:34615)
0810 06:26:02.293110 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:02.293152 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:02.294616 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:02.295137 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:02.295144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:02.295226 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:02.295925 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:02.295948 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.296880 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.296884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:02.298357 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:02.298373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.298880 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.298886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:02.299133 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:02.299832 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:26:02.299836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:02.299897 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:26:02.299987 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:02.300297 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:02.300334 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:02.300343 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 06:26:02.300489 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 06:26:02.300827 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260810 06:26:02.303304 20422 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38689
I20260810 06:26:02.303545 20422 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:02.304080 20422 heartbeater.cc:511] Master 127.4.73.62:38689 requested a full tablet report, sending...
I20260810 06:26:02.305615 20081 ts_manager.cc:194] Registered new tserver with Master: 7d2a6f1a1802455a87a236a85461dac9 (127.4.73.1:46609)
I20260810 06:26:02.306782 20081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.1:34615
W20260810 06:26:02.408216 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:02.408574 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:02.408649 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:02.413625 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:02.413692 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:02.413774 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:26:02.418551 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:02.419587 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:02.421017 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:26:02.431167 20428 server_base.cc:1034] running on GCE node
W20260810 06:26:02.431140 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.431162 20439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.431335 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:02.431849 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:02.432520 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:02.433728 20428 hybrid_clock.cc:648] HybridClock initialized: now 1786343162433697 us; error 41 us; skew 500 ppm
I20260810 06:26:02.436089 20428 webserver.cc:460] Webserver started at http://127.4.73.2:39083/ using document root <none> and password file <none>
I20260810 06:26:02.436730 20428 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:02.436831 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:02.437077 20428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:02.439178 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "3a9abdc2084747d6a90b26ec9b27f75b"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "553f1c1de833c032eba57e869640224d"
server_key_iv: "4cff1316ef0911ae56569e899c6997aa"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.439755 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "3a9abdc2084747d6a90b26ec9b27f75b"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "553f1c1de833c032eba57e869640224d"
server_key_iv: "4cff1316ef0911ae56569e899c6997aa"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.443506 20428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:26:02.446012 20455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.447137 20428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 06:26:02.447268 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "3a9abdc2084747d6a90b26ec9b27f75b"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "553f1c1de833c032eba57e869640224d"
server_key_iv: "4cff1316ef0911ae56569e899c6997aa"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.447381 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:02.468003 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:02.468883 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:02.469084 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:02.469655 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:02.470721 20428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:02.470799 20428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.470886 20428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:02.470937 20428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.480027 20428 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:33491
I20260810 06:26:02.480055 20649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:33491 every 8 connection(s)
I20260810 06:26:02.480978 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:26:02.481643 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 20428
I20260810 06:26:02.481770 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:26:02.482053 4388 external_mini_cluster.cc:1428] Setting key 7f153637c219ea18c18f54acbc6a0867
I20260810 06:26:02.484133 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38689
--builtin_ntp_servers=127.4.73.20:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 06:26:02.491137 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.482958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:38597 (local address 127.4.73.62:38689)
0810 06:26:02.483116 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:02.483121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:02.484738 (+ 1617us) server_negotiation.cc:407] Connection header received
0810 06:26:02.486134 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:02.486138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:02.486191 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:02.486273 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:02.487126 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.487654 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.488946 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.489186 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.489301 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:02.490658 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:02.490661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:02.490663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:02.490675 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:02.490695 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:02.490704 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:02.490764 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:02.490974 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 06:26:02.491025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:26:02.492530 20652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.483317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38689 (local address 127.4.73.2:38597)
0810 06:26:02.484603 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:02.484643 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:02.485932 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:02.486442 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:02.486448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:02.486526 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:02.486989 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:02.487001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.487789 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.487792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:02.488809 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:02.488819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.489374 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.489382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:02.489631 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:02.490339 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:26:02.490344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:02.490405 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:26:02.490493 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:02.490802 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:02.490814 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:02.490823 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 06:26:02.490970 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 06:26:02.491391 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":150,"threads_started":1}
I20260810 06:26:02.493789 20650 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38689
I20260810 06:26:02.494040 20650 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:02.494733 20650 heartbeater.cc:511] Master 127.4.73.62:38689 requested a full tablet report, sending...
I20260810 06:26:02.495725 20081 ts_manager.cc:194] Registered new tserver with Master: 3a9abdc2084747d6a90b26ec9b27f75b (127.4.73.2:33491)
I20260810 06:26:02.496389 20081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.2:38597
W20260810 06:26:02.595372 20653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:02.595685 20653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:02.595754 20653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:02.599619 20653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:02.599710 20653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:02.599805 20653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:26:02.604370 20653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:02.605326 20653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:02.606604 20653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:02.617125 20670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.617166 20663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.617179 20662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:02.617452 20653 server_base.cc:1034] running on GCE node
I20260810 06:26:02.617954 20653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:02.618546 20653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:02.619728 20653 hybrid_clock.cc:648] HybridClock initialized: now 1786343162619704 us; error 34 us; skew 500 ppm
I20260810 06:26:02.621601 20653 webserver.cc:460] Webserver started at http://127.4.73.3:39067/ using document root <none> and password file <none>
I20260810 06:26:02.622167 20653 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:02.622263 20653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:02.622509 20653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:02.624571 20653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "2e3113a4d0f44deb89676b266ff28dd7"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "88e50b118870b73ec041dfe50b163650"
server_key_iv: "53b7a3d202db418b9a783e5b67059aad"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.625145 20653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "2e3113a4d0f44deb89676b266ff28dd7"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "88e50b118870b73ec041dfe50b163650"
server_key_iv: "53b7a3d202db418b9a783e5b67059aad"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.628773 20653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:26:02.631208 20679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.632441 20653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:26:02.632563 20653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "2e3113a4d0f44deb89676b266ff28dd7"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "88e50b118870b73ec041dfe50b163650"
server_key_iv: "53b7a3d202db418b9a783e5b67059aad"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.632668 20653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:02.669006 20653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:02.669766 20653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:02.669968 20653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:02.670550 20653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:02.671525 20653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:02.671598 20653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.671671 20653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:02.671725 20653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.680998 20653 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:46335
I20260810 06:26:02.681038 20893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:46335 every 8 connection(s)
I20260810 06:26:02.681887 20653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:26:02.690706 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.683770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:53147 (local address 127.4.73.62:38689)
0810 06:26:02.683915 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:02.683919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:02.684961 (+ 1042us) server_negotiation.cc:407] Connection header received
0810 06:26:02.686315 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:02.686318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:02.686367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:02.686444 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:02.687305 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.687815 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.688950 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.689157 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.689235 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:02.690265 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:02.690268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:02.690270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:02.690282 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:02.690300 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:02.690308 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:02.690366 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:02.690544 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 06:26:02.690595 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:26:02.691206 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 20653
I20260810 06:26:02.691284 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:26:02.691545 4388 external_mini_cluster.cc:1428] Setting key a2cf213ba25a9d14ea6bf5cf213c1c7a
I20260810 06:26:02.691603 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.684139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38689 (local address 127.4.73.3:53147)
0810 06:26:02.684810 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:02.684851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:02.686117 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:02.686594 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:02.686600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:02.686678 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:02.687180 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:02.687194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.687940 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.687943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:02.688846 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:02.688856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:02.689297 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:02.689300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:02.689450 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:02.690019 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:26:02.690020 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:02.690061 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:26:02.690122 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:02.690384 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:02.690391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:02.690395 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:26:02.690487 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 06:26:02.690691 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":167,"threads_started":1}
I20260810 06:26:02.692770 20895 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38689
I20260810 06:26:02.693017 20895 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:02.693594 20895 heartbeater.cc:511] Master 127.4.73.62:38689 requested a full tablet report, sending...
I20260810 06:26:02.694559 20080 ts_manager.cc:194] Registered new tserver with Master: 2e3113a4d0f44deb89676b266ff28dd7 (127.4.73.3:46335)
I20260810 06:26:02.695116 20080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.3:53147
I20260810 06:26:02.705315 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 06:26:02.708210 20150 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:02.707252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.4.73.62:38689)
0810 06:26:02.707383 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:02.707389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:02.707536 (+ 147us) server_negotiation.cc:407] Connection header received
0810 06:26:02.707626 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:02.707628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:02.707663 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:02.707727 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:02.708098 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32962: BlockingRecv error: recv got EOF from 127.0.0.1:32962 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:26:02.708424 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 20190
I20260810 06:26:02.734961 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 20428
I20260810 06:26:02.750012 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 20653
I20260810 06:26:02.758673 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 20033
2026-08-10T06:26:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (977 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T06:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:26:02Z Disabled control of system clock
I20260810 06:26:02.793028 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43347
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 06:26:02.901818 20914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:02.902108 20914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:02.902145 20914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:02.906147 20914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:26:02.906195 20914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:02.906210 20914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:02.906260 20914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:26:02.906294 20914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:26:02.911077 20914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:02.912140 20914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:02.913543 20914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:02.921784 20927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.921877 20923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:02.921790 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:02.922159 20914 server_base.cc:1034] running on GCE node
I20260810 06:26:02.922740 20914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:02.923799 20914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:02.924980 20914 hybrid_clock.cc:648] HybridClock initialized: now 1786343162924952 us; error 43 us; skew 500 ppm
I20260810 06:26:02.926738 20914 webserver.cc:460] Webserver started at http://127.4.73.62:37361/ using document root <none> and password file <none>
I20260810 06:26:02.927291 20914 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:02.927386 20914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:02.927685 20914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:02.929729 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "7381bce0473e4e8dbed0cc8d6711774a"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "6d358ccbaf07a1b4eea97e009dba3e08"
server_key_iv: "07c68b8f51ca38d3add12264c960385a"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.930286 20914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "7381bce0473e4e8dbed0cc8d6711774a"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "6d358ccbaf07a1b4eea97e009dba3e08"
server_key_iv: "07c68b8f51ca38d3add12264c960385a"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.933765 20914 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:26:02.936007 20937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:02.937131 20914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:26:02.937271 20914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "7381bce0473e4e8dbed0cc8d6711774a"
format_stamp: "Formatted at 2026-08-10 06:26:02 on dist-test-slave-45dx"
server_key: "6d358ccbaf07a1b4eea97e009dba3e08"
server_key_iv: "07c68b8f51ca38d3add12264c960385a"
server_key_version: "encryptionkey@0"
I20260810 06:26:02.937383 20914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:02.941726 20914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:02.942302 20914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:02.942487 20914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:02.949173 20914 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43347
I20260810 06:26:02.949203 21036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43347 every 8 connection(s)
I20260810 06:26:02.950018 20914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:26:02.953047 21038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:02.958472 21038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a: Bootstrap starting.
I20260810 06:26:02.958599 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 20914
I20260810 06:26:02.958700 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:26:02.959007 4388 external_mini_cluster.cc:1428] Setting key 471fa6e1852d8b9ec483542ab7901422
I20260810 06:26:02.961139 21038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:02.961897 21038 log.cc:824] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:02.963891 21038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a: No bootstrap required, opened a new log
I20260810 06:26:02.964586 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:02.960408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.4.73.62:43347)
0810 06:26:02.961029 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:02.961044 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:02.961076 (+ 32us) server_negotiation.cc:407] Connection header received
0810 06:26:02.961752 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:02.961770 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:02.962151 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:02.962486 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:02.963282 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:02.963286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:02.963298 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:02.963308 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:02.963366 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:02.963381 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:02.963507 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:02.963607 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 06:26:02.963836 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260810 06:26:02.967836 21038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7381bce0473e4e8dbed0cc8d6711774a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43347 } }
I20260810 06:26:02.968044 21038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:02.968122 21038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7381bce0473e4e8dbed0cc8d6711774a, State: Initialized, Role: FOLLOWER
I20260810 06:26:02.968568 21038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7381bce0473e4e8dbed0cc8d6711774a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43347 } }
I20260810 06:26:02.968714 21038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:26:02.968777 21038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:26:02.968901 21038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:02.969726 21038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7381bce0473e4e8dbed0cc8d6711774a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43347 } }
I20260810 06:26:02.970064 21038 leader_election.cc:302] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7381bce0473e4e8dbed0cc8d6711774a; no voters:
I20260810 06:26:02.970361 21038 leader_election.cc:288] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:26:02.970484 21046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:26:02.970734 21046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [term 1 LEADER]: Becoming Leader. State: Replica: 7381bce0473e4e8dbed0cc8d6711774a, State: Running, Role: LEADER
I20260810 06:26:02.971029 21046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7381bce0473e4e8dbed0cc8d6711774a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43347 } }
I20260810 06:26:02.971692 21038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:26:02.972880 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7381bce0473e4e8dbed0cc8d6711774a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7381bce0473e4e8dbed0cc8d6711774a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43347 } } }
I20260810 06:26:02.972898 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7381bce0473e4e8dbed0cc8d6711774a. Latest consensus state: current_term: 1 leader_uuid: "7381bce0473e4e8dbed0cc8d6711774a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7381bce0473e4e8dbed0cc8d6711774a" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43347 } } }
I20260810 06:26:02.973037 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [sys.catalog]: This master's current role is: LEADER
I20260810 06:26:02.973042 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a [sys.catalog]: This master's current role is: LEADER
I20260810 06:26:02.973379 21060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:26:02.976358 21060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:26:02.981952 21060 catalog_manager.cc:1347] Generated new cluster ID: 0ba085f97ef3424c94865550402ff5bd
I20260810 06:26:02.982019 21060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:26:03.015548 21060 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:26:03.015944 21060 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:26:03.025717 21060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7381bce0473e4e8dbed0cc8d6711774a: Generated new TSK 0
I20260810 06:26:03.026417 21060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:26:03.032490 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43347
--builtin_ntp_servers=127.4.73.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 06:26:03.145076 21085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:03.145309 21085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:03.145372 21085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:03.149273 21085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:03.149322 21085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:03.149402 21085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:26:03.153931 21085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:03.154927 21085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:03.156255 21085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:03.165824 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:03.165827 21094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:03.165845 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:03.166158 21085 server_base.cc:1034] running on GCE node
I20260810 06:26:03.166548 21085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:03.167193 21085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:03.168376 21085 hybrid_clock.cc:648] HybridClock initialized: now 1786343163168363 us; error 24 us; skew 500 ppm
I20260810 06:26:03.170184 21085 webserver.cc:460] Webserver started at http://127.4.73.1:37757/ using document root <none> and password file <none>
I20260810 06:26:03.170748 21085 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:03.170842 21085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:03.171085 21085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:03.173537 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "a4faa4f9aa8d45c894a6253c567c5dc0"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "9e83ccddf703d38ca11177d93bdf3864"
server_key_iv: "70c34ada5dfadfe97872df925e48fc8a"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.174171 21085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "a4faa4f9aa8d45c894a6253c567c5dc0"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "9e83ccddf703d38ca11177d93bdf3864"
server_key_iv: "70c34ada5dfadfe97872df925e48fc8a"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.177826 21085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 06:26:03.180410 21114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.181557 21085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:26:03.181694 21085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "a4faa4f9aa8d45c894a6253c567c5dc0"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "9e83ccddf703d38ca11177d93bdf3864"
server_key_iv: "70c34ada5dfadfe97872df925e48fc8a"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.181823 21085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:03.186463 21085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:03.187065 21085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:03.187249 21085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:03.187781 21085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:03.188758 21085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:03.188830 21085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.188906 21085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:03.188957 21085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.197891 21085 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:39179
I20260810 06:26:03.197961 21289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:39179 every 8 connection(s)
I20260810 06:26:03.198823 21085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:26:03.199643 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 21085
I20260810 06:26:03.199770 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:26:03.200084 4388 external_mini_cluster.cc:1428] Setting key b4a9e6f7dd29f9a68b3b5df311f5124e
I20260810 06:26:03.202389 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43347
--builtin_ntp_servers=127.4.73.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 06:26:03.206132 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:03.200788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:40893 (local address 127.4.73.62:43347)
0810 06:26:03.200987 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:03.200992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:03.202282 (+ 1290us) server_negotiation.cc:407] Connection header received
0810 06:26:03.204349 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:03.204353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:03.204402 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:03.204501 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:03.205619 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:03.205625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:03.205631 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:03.205648 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:03.205674 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:03.205687 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:03.205789 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:03.205898 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 06:26:03.205946 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 06:26:03.206977 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:03.201264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43347 (local address 127.4.73.1:40893)
0810 06:26:03.202115 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:03.202175 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:03.204034 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:03.204632 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:03.204639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:03.204724 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:03.205341 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:26:03.205343 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:03.205384 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:26:03.205452 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:03.205772 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:03.205777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:03.205780 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:26:03.205916 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 06:26:03.206097 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260810 06:26:03.208248 21290 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43347
I20260810 06:26:03.208544 21290 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:03.208978 21290 heartbeater.cc:511] Master 127.4.73.62:43347 requested a full tablet report, sending...
I20260810 06:26:03.210687 20969 ts_manager.cc:194] Registered new tserver with Master: a4faa4f9aa8d45c894a6253c567c5dc0 (127.4.73.1:39179)
W20260810 06:26:03.318389 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:03.318802 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:03.318856 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:03.323035 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:03.323105 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:03.323163 21296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:26:03.327960 21296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:03.329219 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:03.330634 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:26:03.340626 21296 server_base.cc:1034] running on GCE node
W20260810 06:26:03.340648 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:03.340677 21308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:03.340798 21307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:03.341226 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:03.341930 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:03.343115 21296 hybrid_clock.cc:648] HybridClock initialized: now 1786343163343096 us; error 31 us; skew 500 ppm
I20260810 06:26:03.344872 21296 webserver.cc:460] Webserver started at http://127.4.73.2:43765/ using document root <none> and password file <none>
I20260810 06:26:03.345441 21296 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:03.345538 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:03.345793 21296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:03.347770 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "6e3a0ba1968240c8803b065bccaf7dd8"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "0820f3ad6cabccf741340ed05497fd54"
server_key_iv: "999724b99c8a143bbb8bfa09e836641a"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.348394 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "6e3a0ba1968240c8803b065bccaf7dd8"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "0820f3ad6cabccf741340ed05497fd54"
server_key_iv: "999724b99c8a143bbb8bfa09e836641a"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.351933 21296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:26:03.354390 21326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.355571 21296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:26:03.355705 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "6e3a0ba1968240c8803b065bccaf7dd8"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "0820f3ad6cabccf741340ed05497fd54"
server_key_iv: "999724b99c8a143bbb8bfa09e836641a"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.355816 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:03.360508 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:03.361142 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:03.361336 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:03.361945 21296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:03.362962 21296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:03.363011 21296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.363050 21296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:03.363099 21296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.372247 21296 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:45857
I20260810 06:26:03.372354 21521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:45857 every 8 connection(s)
I20260810 06:26:03.373292 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:26:03.379503 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:03.375202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:44431 (local address 127.4.73.62:43347)
0810 06:26:03.375355 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:03.375360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:03.376577 (+ 1217us) server_negotiation.cc:407] Connection header received
0810 06:26:03.377873 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:03.377875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:03.377913 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:03.377986 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:03.379134 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:03.379137 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:03.379139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:03.379149 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:03.379167 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:03.379174 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:03.379246 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:03.379341 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 06:26:03.379379 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:26:03.380406 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 21296
I20260810 06:26:03.380532 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:26:03.380582 21527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:03.375647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43347 (local address 127.4.73.2:44431)
0810 06:26:03.376429 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:03.376473 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:03.377681 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:03.378122 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:03.378128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:03.378215 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:03.378878 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:26:03.378879 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:03.378923 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:26:03.378995 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:03.379230 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:03.379234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:03.379238 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:26:03.379364 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 06:26:03.379555 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
I20260810 06:26:03.380847 4388 external_mini_cluster.cc:1428] Setting key 220ad9874681e6dd6b1e24fa7ebdd77e
I20260810 06:26:03.381946 21523 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43347
I20260810 06:26:03.382184 21523 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:03.382640 21523 heartbeater.cc:511] Master 127.4.73.62:43347 requested a full tablet report, sending...
I20260810 06:26:03.383003 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43347
--builtin_ntp_servers=127.4.73.20:36919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 06:26:03.383687 20969 ts_manager.cc:194] Registered new tserver with Master: 6e3a0ba1968240c8803b065bccaf7dd8 (127.4.73.2:45857)
W20260810 06:26:03.497850 21528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:03.498095 21528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:03.498131 21528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:03.502017 21528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:03.502064 21528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:03.502115 21528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:26:03.506556 21528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:36919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:03.507458 21528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:03.508751 21528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:03.517864 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:03.517856 21537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:03.517881 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:03.518066 21528 server_base.cc:1034] running on GCE node
I20260810 06:26:03.518618 21528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:03.519254 21528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:03.520378 21528 hybrid_clock.cc:648] HybridClock initialized: now 1786343163520346 us; error 44 us; skew 500 ppm
I20260810 06:26:03.522223 21528 webserver.cc:460] Webserver started at http://127.4.73.3:41049/ using document root <none> and password file <none>
I20260810 06:26:03.522768 21528 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:03.522827 21528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:03.523062 21528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:03.525087 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "5a2d810d3aad4ec0afcb51638e545b1b"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "44e7a94ca2bc2c5bbd5ccc28c802582b"
server_key_iv: "db104052ff54c18d2e09f3e3f74fed9f"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.525688 21528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "5a2d810d3aad4ec0afcb51638e545b1b"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "44e7a94ca2bc2c5bbd5ccc28c802582b"
server_key_iv: "db104052ff54c18d2e09f3e3f74fed9f"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.529346 21528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:26:03.531941 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.533125 21528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 06:26:03.533272 21528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "5a2d810d3aad4ec0afcb51638e545b1b"
format_stamp: "Formatted at 2026-08-10 06:26:03 on dist-test-slave-45dx"
server_key: "44e7a94ca2bc2c5bbd5ccc28c802582b"
server_key_iv: "db104052ff54c18d2e09f3e3f74fed9f"
server_key_version: "encryptionkey@0"
I20260810 06:26:03.533416 21528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:03.537937 21528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:03.538513 21528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:03.538692 21528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:03.539212 21528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:03.540184 21528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:03.540253 21528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.540350 21528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:03.540405 21528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:03.548835 21528 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:35631
I20260810 06:26:03.548893 21741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:35631 every 8 connection(s)
I20260810 06:26:03.549769 21528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:26:03.550981 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 21528
I20260810 06:26:03.551100 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:26:03.551376 4388 external_mini_cluster.cc:1428] Setting key 6ecd8366889606719776e602e2287201
I20260810 06:26:03.555876 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:03.551540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:35919 (local address 127.4.73.62:43347)
0810 06:26:03.551707 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:03.551711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:03.552800 (+ 1089us) server_negotiation.cc:407] Connection header received
0810 06:26:03.554242 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:03.554245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:03.554307 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:03.554381 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:03.555388 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:03.555391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:03.555401 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:26:03.555412 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:03.555434 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:26:03.555448 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:03.555538 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:03.555702 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 06:26:03.555745 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:26:03.556809 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:03.551948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43347 (local address 127.4.73.3:35919)
0810 06:26:03.552663 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:03.552710 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:03.554031 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:03.554548 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:03.554555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:03.554634 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:03.555145 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:26:03.555146 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:03.555184 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:26:03.555248 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:03.555527 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:03.555536 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:03.555542 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:26:03.555743 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 06:26:03.555995 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":124,"threads_started":1}
I20260810 06:26:03.557866 21742 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43347
I20260810 06:26:03.558101 21742 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:03.558493 21742 heartbeater.cc:511] Master 127.4.73.62:43347 requested a full tablet report, sending...
I20260810 06:26:03.559511 20969 ts_manager.cc:194] Registered new tserver with Master: 5a2d810d3aad4ec0afcb51638e545b1b (127.4.73.3:35631)
I20260810 06:26:03.565191 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 06:26:03.568281 21043 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:03.567319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.4.73.62:43347)
0810 06:26:03.567474 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:03.567479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:03.567658 (+ 179us) server_negotiation.cc:407] Connection header received
0810 06:26:03.567819 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:03.567822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:03.567857 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:03.567941 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:03.568186 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47370: BlockingRecv error: recv got EOF from 127.0.0.1:47370 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:26:03.568601 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 21085
I20260810 06:26:03.581877 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 21296
I20260810 06:26:03.592571 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 21528
I20260810 06:26:03.603991 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 20914
2026-08-10T06:26:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (847 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:26:03 dist-test-slave-45dx krb5kdc[21757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:26:03 dist-test-slave-45dx krb5kdc[21757](info): set up 2 sockets
Aug 10 06:26:03 dist-test-slave-45dx krb5kdc[21757](info): commencing operation
krb5kdc: starting...
W20260810 06:26:04.871642 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:26:04 dist-test-slave-45dx krb5kdc[21757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:26:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:26:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:26:04.997381 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43405
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:26:05.106856 21779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:05.107102 21779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:05.107141 21779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:05.110884 21779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:26:05.110931 21779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:05.110950 21779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:05.110966 21779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:26:05.110981 21779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:26:05.115931 21779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:05.116863 21779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:05.118173 21779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:05.129165 21785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:05.129135 21784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:05.129135 21788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:05.130024 21779 server_base.cc:1034] running on GCE node
I20260810 06:26:05.130695 21779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:05.131907 21779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:05.133093 21779 hybrid_clock.cc:648] HybridClock initialized: now 1786343165133082 us; error 37 us; skew 500 ppm
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:05.135690 21779 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:26:05.136624 21779 webserver.cc:460] Webserver started at http://127.4.73.62:46625/ using document root <none> and password file <none>
I20260810 06:26:05.137164 21779 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:05.137252 21779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:05.137473 21779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:05.139382 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "547779dcd3af4d7ca96fa4fee7de175f"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "7889b192acffbd609b0ef22974355864"
server_key_iv: "e322a7e296b0bcc4741e1bff7a5dc2dd"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.139957 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "547779dcd3af4d7ca96fa4fee7de175f"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "7889b192acffbd609b0ef22974355864"
server_key_iv: "e322a7e296b0bcc4741e1bff7a5dc2dd"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.143570 21779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:26:05.145921 21798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.147053 21779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:26:05.147189 21779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "547779dcd3af4d7ca96fa4fee7de175f"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "7889b192acffbd609b0ef22974355864"
server_key_iv: "e322a7e296b0bcc4741e1bff7a5dc2dd"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.147294 21779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:05.178143 21779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:05.181129 21779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:05.181329 21779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:05.188230 21779 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43405
I20260810 06:26:05.188536 21887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43405 every 8 connection(s)
I20260810 06:26:05.189208 21779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:26:05.191912 21888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:05.193564 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 21779
I20260810 06:26:05.193699 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:26:05.193950 4388 external_mini_cluster.cc:1428] Setting key 52a39bb886d5974ab124d8035e1f724e
I20260810 06:26:05.198073 21888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f: Bootstrap starting.
I20260810 06:26:05.200639 21888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:05.201577 21888 log.cc:824] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:05.203521 21888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f: No bootstrap required, opened a new log
I20260810 06:26:05.207037 21888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547779dcd3af4d7ca96fa4fee7de175f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43405 } }
I20260810 06:26:05.207279 21888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:05.207360 21888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547779dcd3af4d7ca96fa4fee7de175f, State: Initialized, Role: FOLLOWER
I20260810 06:26:05.207779 21888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547779dcd3af4d7ca96fa4fee7de175f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43405 } }
I20260810 06:26:05.207855 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.195156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.4.73.62:43405)
0810 06:26:05.195549 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.195564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.195599 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:26:05.196190 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.196205 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.196573 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.196845 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:05.197740 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.198725 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.199850 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.200414 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.200535 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.203275 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:05.203286 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:05.203298 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:05.203330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:05.204960 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.205409 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.205414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.205418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.205526 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.205817 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.205823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.205826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.206194 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:05.206432 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.206703 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.206982 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
I20260810 06:26:05.207986 21888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:26:05.208066 21888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:26:05.208189 21888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:05.209270 21888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547779dcd3af4d7ca96fa4fee7de175f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43405 } }
I20260810 06:26:05.209602 21888 leader_election.cc:302] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547779dcd3af4d7ca96fa4fee7de175f; no voters:
I20260810 06:26:05.209939 21888 leader_election.cc:288] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:26:05.210052 21894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:26:05.210268 21894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [term 1 LEADER]: Becoming Leader. State: Replica: 547779dcd3af4d7ca96fa4fee7de175f, State: Running, Role: LEADER
I20260810 06:26:05.210644 21894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547779dcd3af4d7ca96fa4fee7de175f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43405 } }
I20260810 06:26:05.211162 21888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:26:05.212260 21898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547779dcd3af4d7ca96fa4fee7de175f. Latest consensus state: current_term: 1 leader_uuid: "547779dcd3af4d7ca96fa4fee7de175f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547779dcd3af4d7ca96fa4fee7de175f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43405 } } }
I20260810 06:26:05.212447 21898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [sys.catalog]: This master's current role is: LEADER
I20260810 06:26:05.212415 21897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547779dcd3af4d7ca96fa4fee7de175f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547779dcd3af4d7ca96fa4fee7de175f" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43405 } } }
I20260810 06:26:05.212512 21897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f [sys.catalog]: This master's current role is: LEADER
W20260810 06:26:05.216099 21922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:26:05.216177 21922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:26:05.216271 21911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:26:05.217021 21911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:26:05.221838 21911 catalog_manager.cc:1347] Generated new cluster ID: 4cfffff89f26426398e2818c8421368c
I20260810 06:26:05.221920 21911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:26:05.231994 21911 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:26:05.233094 21911 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:26:05.236864 21911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f: Generated new TSK 0
I20260810 06:26:05.237421 21911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:26:05.272570 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43405
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:26:05.384454 21932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:05.384722 21932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:05.384795 21932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:05.388690 21932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:05.388768 21932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:05.388856 21932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:26:05.393981 21932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:05.395027 21932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:05.396055 21932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:05.406044 21940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:05.406069 21943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:05.406101 21932 server_base.cc:1034] running on GCE node
W20260810 06:26:05.406044 21945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:05.406754 21932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:05.407389 21932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:05.408560 21932 hybrid_clock.cc:648] HybridClock initialized: now 1786343165408540 us; error 31 us; skew 500 ppm
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:05.411270 21932 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:26:05.412398 21932 webserver.cc:460] Webserver started at http://127.4.73.1:45991/ using document root <none> and password file <none>
I20260810 06:26:05.412976 21932 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:05.413038 21932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:05.413300 21932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:05.415300 21932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "636097dd51e745daa53208b8531f87ce"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "56d3eaae2e5158baceecf85b304c6644"
server_key_iv: "699e6da1962676d98c3116e55bb4b400"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.415864 21932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "636097dd51e745daa53208b8531f87ce"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "56d3eaae2e5158baceecf85b304c6644"
server_key_iv: "699e6da1962676d98c3116e55bb4b400"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.419445 21932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:26:05.421983 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.423635 21932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 06:26:05.423789 21932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "636097dd51e745daa53208b8531f87ce"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "56d3eaae2e5158baceecf85b304c6644"
server_key_iv: "699e6da1962676d98c3116e55bb4b400"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.423918 21932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:05.434288 21932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:05.437077 21932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:05.437271 21932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:05.437855 21932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:05.438639 21932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:05.438710 21932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.438791 21932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:05.438846 21932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.448863 21932 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:35293
I20260810 06:26:05.448899 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:35293 every 8 connection(s)
I20260810 06:26:05.449734 21932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:26:05.459017 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 21932
I20260810 06:26:05.459151 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:05.459448 4388 external_mini_cluster.cc:1428] Setting key 7cf9c084047b7290e4c6d2711a664c6e
I20260810 06:26:05.464140 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.451692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:42119 (local address 127.4.73.62:43405)
0810 06:26:05.451845 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.451850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.452713 (+ 863us) server_negotiation.cc:407] Connection header received
0810 06:26:05.454083 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.454087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.454144 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.454251 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:05.455393 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.455969 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.457143 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.457386 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.457499 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.460183 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:05.460186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:05.460188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:05.460232 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:05.461766 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.462428 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.462430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.462431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.462525 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.462935 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.462937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.462938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.463157 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:05.463265 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.463786 (+ 521us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.463929 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:26:05.464818 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.452058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43405 (local address 127.4.73.1:42119)
0810 06:26:05.452555 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:05.452613 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:05.453878 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:05.454428 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:05.454434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:05.454799 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:05.455246 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.455258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.456107 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.456111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.457018 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.457027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.457536 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.457540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.457703 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.458225 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:05.458240 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:05.459984 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:05.461890 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:05.461899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:05.461915 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:05.462333 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:05.462616 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:05.462622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:05.462626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:05.462837 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:05.463282 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:05.463292 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:05.463559 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 06:26:05.463710 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 06:26:05.463946 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":114,"threads_started":1}
I20260810 06:26:05.466002 22164 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43405
I20260810 06:26:05.466259 22164 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:05.466827 22164 heartbeater.cc:511] Master 127.4.73.62:43405 requested a full tablet report, sending...
I20260810 06:26:05.468506 21827 ts_manager.cc:194] Registered new tserver with Master: 636097dd51e745daa53208b8531f87ce (127.4.73.1:35293)
I20260810 06:26:05.469595 21827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:42119
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:26:05.487435 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43405
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:26:05.597579 22170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:05.597860 22170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:05.597944 22170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:05.601910 22170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:05.601994 22170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:05.602074 22170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:26:05.607267 22170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:05.608220 22170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:05.609264 22170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:05.619657 22180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:05.619686 22186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:05.619800 22170 server_base.cc:1034] running on GCE node
W20260810 06:26:05.619657 22181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:05.620621 22170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:05.621261 22170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:05.622460 22170 hybrid_clock.cc:648] HybridClock initialized: now 1786343165622442 us; error 32 us; skew 500 ppm
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:05.625304 22170 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:26:05.626166 22170 webserver.cc:460] Webserver started at http://127.4.73.2:40071/ using document root <none> and password file <none>
I20260810 06:26:05.626737 22170 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:05.626901 22170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:05.627103 22170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:05.629191 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "45d35404176246ea90fadb7fe8bc879f"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "16a45f5ee93db0f07b7ebfb3008993e3"
server_key_iv: "a95bd1d57127cb3d9fb608c83d41f21b"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.629719 22170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "45d35404176246ea90fadb7fe8bc879f"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "16a45f5ee93db0f07b7ebfb3008993e3"
server_key_iv: "a95bd1d57127cb3d9fb608c83d41f21b"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.633363 22170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:26:05.635769 22203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.637030 22170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:26:05.637166 22170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "45d35404176246ea90fadb7fe8bc879f"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "16a45f5ee93db0f07b7ebfb3008993e3"
server_key_iv: "a95bd1d57127cb3d9fb608c83d41f21b"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.637303 22170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:05.650756 22170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:05.653772 22170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:05.653972 22170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:05.654552 22170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:05.655365 22170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:05.655445 22170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.655515 22170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:05.655570 22170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.664728 22170 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:33901
I20260810 06:26:05.664788 22383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:33901 every 8 connection(s)
I20260810 06:26:05.665632 22170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:26:05.674404 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 22170
I20260810 06:26:05.674543 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:26:05.674932 4388 external_mini_cluster.cc:1428] Setting key 3c8e7574c3179ada515495992aa3b9c9
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:05.679934 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.667458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:48527 (local address 127.4.73.62:43405)
0810 06:26:05.667590 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.667593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.668503 (+ 910us) server_negotiation.cc:407] Connection header received
0810 06:26:05.669889 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.669892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.669948 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.670033 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:05.671144 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.671679 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.672886 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.673115 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.673229 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.676056 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:05.676060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:05.676063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:05.676086 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:05.677760 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.678266 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.678272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.678277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.678386 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.678820 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.678825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.678827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.679024 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:05.679144 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.679628 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.679755 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:26:05.680876 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.667821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43405 (local address 127.4.73.2:48527)
0810 06:26:05.668342 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:05.668404 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:05.669668 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:05.670200 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:05.670206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:05.670542 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:05.670992 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.671006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.671807 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.671812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.672744 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.672754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.673270 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.673274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.673446 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.674045 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:05.674062 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:05.675898 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:05.677897 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:05.677902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:05.677912 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:05.678166 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:05.678539 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:05.678542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:05.678544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:05.678687 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:05.679143 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:05.679150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:05.679390 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 06:26:05.679584 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 06:26:05.679912 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260810 06:26:05.682080 22384 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43405
I20260810 06:26:05.682358 22384 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:05.683005 22384 heartbeater.cc:511] Master 127.4.73.62:43405 requested a full tablet report, sending...
I20260810 06:26:05.684237 21827 ts_manager.cc:194] Registered new tserver with Master: 45d35404176246ea90fadb7fe8bc879f (127.4.73.2:33901)
I20260810 06:26:05.685101 21827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:48527
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:26:05.705240 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43405
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:26:05.819171 22391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:05.819417 22391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:05.819459 22391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:05.823539 22391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:05.823593 22391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:05.823649 22391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:26:05.828956 22391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:05.829918 22391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:05.830937 22391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:05.840762 22398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:05.840749 22405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:05.840826 22391 server_base.cc:1034] running on GCE node
W20260810 06:26:05.840749 22396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:05.841545 22391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:05.842232 22391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:05.843405 22391 hybrid_clock.cc:648] HybridClock initialized: now 1786343165843387 us; error 33 us; skew 500 ppm
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:05.846356 22391 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:26:05.847250 22391 webserver.cc:460] Webserver started at http://127.4.73.3:43551/ using document root <none> and password file <none>
I20260810 06:26:05.847851 22391 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:05.847954 22391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:05.848197 22391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:26:05.850291 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "381acc849a7a401db33057ac3002dbdb"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "e859291b1c59cc6a4aead5ba4342629b"
server_key_iv: "08205219da86411c126eccaa0abc854e"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.850944 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "381acc849a7a401db33057ac3002dbdb"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "e859291b1c59cc6a4aead5ba4342629b"
server_key_iv: "08205219da86411c126eccaa0abc854e"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.854607 22391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:26:05.857110 22415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.858310 22391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:26:05.858448 22391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "381acc849a7a401db33057ac3002dbdb"
format_stamp: "Formatted at 2026-08-10 06:26:05 on dist-test-slave-45dx"
server_key: "e859291b1c59cc6a4aead5ba4342629b"
server_key_iv: "08205219da86411c126eccaa0abc854e"
server_key_version: "encryptionkey@0"
I20260810 06:26:05.858562 22391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:05.876572 22391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:05.879882 22391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:05.880098 22391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:05.880755 22391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:05.881590 22391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:05.881664 22391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.881731 22391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:05.881791 22391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:05.890305 22391 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:37885
I20260810 06:26:05.890350 22594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:37885 every 8 connection(s)
I20260810 06:26:05.891157 22391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:26:05.892282 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 22391
I20260810 06:26:05.892422 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:26:05.892735 4388 external_mini_cluster.cc:1428] Setting key c27303313673e64060c0ff90696848b1
Aug 10 06:26:05 dist-test-slave-45dx krb5kdc[21757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343165, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:05.904644 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.893142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:49293 (local address 127.4.73.62:43405)
0810 06:26:05.893348 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.893353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.894250 (+ 897us) server_negotiation.cc:407] Connection header received
0810 06:26:05.895686 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.895689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.895736 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.895806 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:05.896958 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.897493 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.898645 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.898859 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.898937 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.901575 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:05.901578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:05.901581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:05.901604 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:05.902858 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.903310 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.903312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.903313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.903393 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.903681 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.903683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.903684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.903849 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:05.903924 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.904428 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.904525 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 06:26:05.905436 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.893628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43405 (local address 127.4.73.3:49293)
0810 06:26:05.894116 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:05.894169 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:05.895475 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:05.895939 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:05.895945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:05.896299 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:05.896834 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.896848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.897620 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.897632 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.898540 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.898550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.898987 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.898990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.899132 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.899854 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:05.899870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:05.901424 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:05.902960 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:05.902964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:05.902975 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:05.903224 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:05.903471 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:05.903473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:05.903474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:05.903602 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:05.903936 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:05.903940 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:05.904154 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 06:26:05.904360 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 06:26:05.904620 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260810 06:26:05.906759 22596 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43405
I20260810 06:26:05.907001 22596 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:05.907509 22596 heartbeater.cc:511] Master 127.4.73.62:43405 requested a full tablet report, sending...
I20260810 06:26:05.908530 21827 ts_manager.cc:194] Registered new tserver with Master: 381acc849a7a401db33057ac3002dbdb (127.4.73.3:37885)
I20260810 06:26:05.909047 21827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:49293
I20260810 06:26:05.918735 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:26:05.928638 21891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.920827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.4.73.62:43405)
0810 06:26:05.920978 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.920983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.921134 (+ 151us) server_negotiation.cc:407] Connection header received
0810 06:26:05.921306 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.921309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.921369 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.921457 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:05.922222 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.922746 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.923639 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.923863 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.923972 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.925383 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:05.925387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:05.925390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:05.925423 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:05.926727 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.927173 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.927177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.927179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.927266 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:05.927554 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:05.927558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:05.927559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:05.927883 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:05.928028 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.928278 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.928447 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:26:05.931635 21827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:26:05.935065 21827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:26:05.949599 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.943835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.4.73.3:37885)
0810 06:26:05.944395 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.944414 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.944436 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:26:05.944533 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.944538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.944750 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.944912 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:05.945469 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.946476 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.947951 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.948850 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.948987 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.949142 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.949357 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.949452 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":161,"threads_started":1}
I20260810 06:26:05.949589 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.943680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:37885 (local address 127.0.0.1:44658)
0810 06:26:05.944047 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:05.944095 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:05.944218 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:05.945114 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:05.945121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:05.945150 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:05.945298 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.945305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.946596 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.946602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.947779 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.947788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.949052 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.949055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.949158 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.949198 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 06:26:05.949325 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 06:26:05.949450 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"spinlock_wait_cycles":2304,"thread_start_us":144,"threads_started":1}
I20260810 06:26:05.950592 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.944885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35293 (local address 127.0.0.1:49758)
0810 06:26:05.945105 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:05.945118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:05.945191 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:05.946075 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:05.946078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:05.946098 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:05.946188 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.946194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.947590 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.947594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.948892 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.948905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.950190 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.950193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.950283 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.950314 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 06:26:05.950386 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 06:26:05.950455 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260810 06:26:05.950958 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.944733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.4.73.2:33901)
0810 06:26:05.946330 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.946346 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.946367 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:26:05.946443 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.946448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.946616 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.946764 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:05.947075 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.948146 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.949480 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.950276 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.950391 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.950521 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.950725 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.950813 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":151,"threads_started":1}
I20260810 06:26:05.950955 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.944509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:33901 (local address 127.0.0.1:57710)
0810 06:26:05.944755 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:05.944769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:05.944870 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:05.946754 (+ 1884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:05.946757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:05.946776 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:05.946947 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.946955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.948362 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.948365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.949325 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:05.949337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.950402 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.950406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:05.950504 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.950527 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:26:05.950605 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:26:05.950674 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":61,"thread_start_us":120,"threads_started":1}
I20260810 06:26:05.951370 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:05.945305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.4.73.1:35293)
0810 06:26:05.945702 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:05.945714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:05.945731 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:26:05.945793 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:05.945796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:05.945940 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:05.946053 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:05.946327 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.947448 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.949044 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:05.949836 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:05.949965 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:05.950118 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:05.950886 (+ 768us) server_negotiation.cc:299] Negotiation successful
0810 06:26:05.951222 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":121,"threads_started":1}
I20260810 06:26:05.953040 22492 tablet_service.cc:1467] Processing CreateTablet for tablet e508db2e765748388fdb5de91eaf91e0 (DEFAULT_TABLE table=test-table [id=a490fdadc19445ce840d4a4171b1b430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:26:05.953711 22284 tablet_service.cc:1467] Processing CreateTablet for tablet e508db2e765748388fdb5de91eaf91e0 (DEFAULT_TABLE table=test-table [id=a490fdadc19445ce840d4a4171b1b430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:26:05.954195 22492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e508db2e765748388fdb5de91eaf91e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:05.954116 22050 tablet_service.cc:1467] Processing CreateTablet for tablet e508db2e765748388fdb5de91eaf91e0 (DEFAULT_TABLE table=test-table [id=a490fdadc19445ce840d4a4171b1b430]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:26:05.954635 22284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e508db2e765748388fdb5de91eaf91e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:05.955049 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e508db2e765748388fdb5de91eaf91e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:05.960762 22622 tablet_bootstrap.cc:492] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Bootstrap starting.
I20260810 06:26:05.961736 22624 tablet_bootstrap.cc:492] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Bootstrap starting.
I20260810 06:26:05.963557 22622 tablet_bootstrap.cc:654] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:05.963661 22624 tablet_bootstrap.cc:654] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:05.964300 22622 log.cc:824] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:05.964638 22625 tablet_bootstrap.cc:492] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: Bootstrap starting.
I20260810 06:26:05.964803 22624 log.cc:824] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:05.966624 22625 tablet_bootstrap.cc:654] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:05.966626 22624 tablet_bootstrap.cc:492] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: No bootstrap required, opened a new log
I20260810 06:26:05.966840 22624 ts_tablet_manager.cc:1397] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:26:05.967221 22625 log.cc:824] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:05.968839 22625 tablet_bootstrap.cc:492] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: No bootstrap required, opened a new log
I20260810 06:26:05.969038 22625 ts_tablet_manager.cc:1397] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:26:05.969955 22624 raft_consensus.cc:348] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:05.970194 22624 raft_consensus.cc:374] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:05.970278 22624 raft_consensus.cc:729] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381acc849a7a401db33057ac3002dbdb, State: Initialized, Role: FOLLOWER
I20260810 06:26:05.970719 22624 consensus_queue.cc:260] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:05.971406 22596 heartbeater.cc:503] Master 127.4.73.62:43405 was elected leader, sending a full tablet report...
I20260810 06:26:05.971661 22624 ts_tablet_manager.cc:1428] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:26:05.972713 22622 tablet_bootstrap.cc:492] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: No bootstrap required, opened a new log
I20260810 06:26:05.972940 22622 ts_tablet_manager.cc:1397] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260810 06:26:05.973275 22625 raft_consensus.cc:348] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:05.973548 22625 raft_consensus.cc:374] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:05.973608 22625 raft_consensus.cc:729] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636097dd51e745daa53208b8531f87ce, State: Initialized, Role: FOLLOWER
I20260810 06:26:05.974071 22625 consensus_queue.cc:260] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:05.974877 22164 heartbeater.cc:503] Master 127.4.73.62:43405 was elected leader, sending a full tablet report...
I20260810 06:26:05.975057 22625 ts_tablet_manager.cc:1428] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:26:05.977061 22622 raft_consensus.cc:348] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:05.977265 22622 raft_consensus.cc:374] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:05.977325 22622 raft_consensus.cc:729] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d35404176246ea90fadb7fe8bc879f, State: Initialized, Role: FOLLOWER
I20260810 06:26:05.977815 22622 consensus_queue.cc:260] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:05.978508 22384 heartbeater.cc:503] Master 127.4.73.62:43405 was elected leader, sending a full tablet report...
I20260810 06:26:05.978806 22622 ts_tablet_manager.cc:1428] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 06:26:06.142088 22597 tablet.cc:2369] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:26:06.166867 22386 tablet.cc:2369] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:26:06.201107 22165 tablet.cc:2369] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:26:06.316756 22630 raft_consensus.cc:482] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:26:06.317059 22630 raft_consensus.cc:504] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:06.318364 22630 leader_election.cc:288] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45d35404176246ea90fadb7fe8bc879f (127.4.73.2:33901), 636097dd51e745daa53208b8531f87ce (127.4.73.1:35293)
I20260810 06:26:06.322297 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:06.318626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:33901 (local address 127.4.73.3:35899)
0810 06:26:06.318773 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:06.318790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:06.318920 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:06.319246 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:06.319249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:06.319269 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:06.319394 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:06.319400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.320403 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.320407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:06.321199 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:06.321208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.321868 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.321871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:06.321977 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:06.322003 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:26:06.322092 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 06:26:06.322163 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 06:26:06.322300 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:06.318741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:35899 (local address 127.4.73.2:33901)
0810 06:26:06.318935 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:06.318942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:06.319003 (+ 61us) server_negotiation.cc:407] Connection header received
0810 06:26:06.319065 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:06.319070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:06.319132 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:06.319228 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:06.319513 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.320239 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.321338 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.321734 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.321866 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:06.321930 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:06.322109 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 06:26:06.322179 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 06:26:06.322717 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:06.318980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35293 (local address 127.4.73.3:36171)
0810 06:26:06.319439 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:06.319450 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:06.319570 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:06.319836 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:06.319839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:06.319856 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:06.319985 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:06.319990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.320823 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.320827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:06.321586 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:06.321595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.322249 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.322253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:06.322360 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:06.322391 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 06:26:06.322509 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 06:26:06.322569 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":165,"threads_started":1}
I20260810 06:26:06.322743 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:06.319165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:36171 (local address 127.4.73.1:35293)
0810 06:26:06.319317 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:06.319321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:06.319534 (+ 213us) server_negotiation.cc:407] Connection header received
0810 06:26:06.319683 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:06.319685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:06.319726 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:06.319816 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:06.320116 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.320700 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.321717 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.322125 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.322245 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:06.322312 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:06.322543 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 06:26:06.322623 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:26:06.323001 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e508db2e765748388fdb5de91eaf91e0" candidate_uuid: "381acc849a7a401db33057ac3002dbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d35404176246ea90fadb7fe8bc879f" is_pre_election: true
I20260810 06:26:06.323315 22312 raft_consensus.cc:2454] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 381acc849a7a401db33057ac3002dbdb in term 0.
I20260810 06:26:06.323457 22085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e508db2e765748388fdb5de91eaf91e0" candidate_uuid: "381acc849a7a401db33057ac3002dbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636097dd51e745daa53208b8531f87ce" is_pre_election: true
I20260810 06:26:06.323756 22085 raft_consensus.cc:2454] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 381acc849a7a401db33057ac3002dbdb in term 0.
I20260810 06:26:06.323776 22417 leader_election.cc:302] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381acc849a7a401db33057ac3002dbdb, 45d35404176246ea90fadb7fe8bc879f; no voters:
I20260810 06:26:06.324112 22630 raft_consensus.cc:2781] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:26:06.324189 22630 raft_consensus.cc:482] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:26:06.324219 22630 raft_consensus.cc:3037] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:06.325300 22630 raft_consensus.cc:504] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:06.325628 22630 leader_election.cc:288] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [CANDIDATE]: Term 1 election: Requested vote from peers 45d35404176246ea90fadb7fe8bc879f (127.4.73.2:33901), 636097dd51e745daa53208b8531f87ce (127.4.73.1:35293)
I20260810 06:26:06.326001 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e508db2e765748388fdb5de91eaf91e0" candidate_uuid: "381acc849a7a401db33057ac3002dbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45d35404176246ea90fadb7fe8bc879f"
I20260810 06:26:06.326082 22085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e508db2e765748388fdb5de91eaf91e0" candidate_uuid: "381acc849a7a401db33057ac3002dbdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "636097dd51e745daa53208b8531f87ce"
I20260810 06:26:06.326133 22312 raft_consensus.cc:3037] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:06.326227 22085 raft_consensus.cc:3037] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:06.327270 22312 raft_consensus.cc:2454] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381acc849a7a401db33057ac3002dbdb in term 1.
I20260810 06:26:06.327291 22085 raft_consensus.cc:2454] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381acc849a7a401db33057ac3002dbdb in term 1.
I20260810 06:26:06.327541 22417 leader_election.cc:302] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381acc849a7a401db33057ac3002dbdb, 636097dd51e745daa53208b8531f87ce; no voters:
I20260810 06:26:06.327754 22630 raft_consensus.cc:2781] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:26:06.327994 22630 raft_consensus.cc:686] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 1 LEADER]: Becoming Leader. State: Replica: 381acc849a7a401db33057ac3002dbdb, State: Running, Role: LEADER
I20260810 06:26:06.328361 22630 consensus_queue.cc:237] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } }
I20260810 06:26:06.331380 21826 catalog_manager.cc:5495] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 381acc849a7a401db33057ac3002dbdb (127.4.73.3). New cstate: current_term: 1 leader_uuid: "381acc849a7a401db33057ac3002dbdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "381acc849a7a401db33057ac3002dbdb" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 } health_report { overall_health: UNKNOWN } } }
I20260810 06:26:06.400938 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:06.397330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.4.73.3:37885)
0810 06:26:06.397469 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:06.397473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:06.397486 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:26:06.397647 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:06.397650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:06.397712 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:06.397814 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:26:06.398111 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.398627 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.399501 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:06.399711 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:06.399791 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:06.400138 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:26:06.400585 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:26:06.400677 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:06.400762 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 06:26:06.400813 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:26:06.407034 22312 raft_consensus.cc:1261] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 1 FOLLOWER]: Refusing update from remote peer 381acc849a7a401db33057ac3002dbdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:26:06.407119 22085 raft_consensus.cc:1261] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 1 FOLLOWER]: Refusing update from remote peer 381acc849a7a401db33057ac3002dbdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:26:06.407863 22642 consensus_queue.cc:1035] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45d35404176246ea90fadb7fe8bc879f" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:26:06.408111 22630 consensus_queue.cc:1035] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "636097dd51e745daa53208b8531f87ce" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:26:06.415393 22648 mvcc.cc:205] Tried to move back new op lower bound from 7316861609595973632 to 7316861609283366912. Current Snapshot: MvccSnapshot[applied={T|T < 7316861609595973632}]
I20260810 06:26:06.416926 22650 mvcc.cc:205] Tried to move back new op lower bound from 7316861609595973632 to 7316861609283366912. Current Snapshot: MvccSnapshot[applied={T|T < 7316861609595973632}]
I20260810 06:26:06.422631 21827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:26:06.422794 21827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:26:06.425195 21827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 547779dcd3af4d7ca96fa4fee7de175f: Sending DeleteTablet for 3 replicas of tablet e508db2e765748388fdb5de91eaf91e0
I20260810 06:26:06.426287 22492 tablet_service.cc:1514] Processing DeleteTablet for tablet e508db2e765748388fdb5de91eaf91e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:26:06 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:44658
I20260810 06:26:06.426368 22284 tablet_service.cc:1514] Processing DeleteTablet for tablet e508db2e765748388fdb5de91eaf91e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:26:06 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:57710
I20260810 06:26:06.426952 22664 tablet_replica.cc:330] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: stopping tablet replica
I20260810 06:26:06.427243 22664 raft_consensus.cc:2229] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:26:06.427553 22664 raft_consensus.cc:2258] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:26:06.427366 22050 tablet_service.cc:1514] Processing DeleteTablet for tablet e508db2e765748388fdb5de91eaf91e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:26:06 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:49758
I20260810 06:26:06.428948 22666 tablet_replica.cc:330] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce: stopping tablet replica
I20260810 06:26:06.429235 22666 raft_consensus.cc:2229] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:26:06.429253 22664 ts_tablet_manager.cc:1905] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:26:06.429452 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 21932
I20260810 06:26:06.429502 22666 raft_consensus.cc:2258] T e508db2e765748388fdb5de91eaf91e0 P 636097dd51e745daa53208b8531f87ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:26:06.432870 22664 ts_tablet_manager.cc:1918] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:26:06.433007 22664 log.cc:1196] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/e508db2e765748388fdb5de91eaf91e0
I20260810 06:26:06.433372 22664 ts_tablet_manager.cc:1939] T e508db2e765748388fdb5de91eaf91e0 P 45d35404176246ea90fadb7fe8bc879f: Deleting consensus metadata
I20260810 06:26:06.435412 21799 catalog_manager.cc:4841] TS 45d35404176246ea90fadb7fe8bc879f (127.4.73.2:33901): tablet e508db2e765748388fdb5de91eaf91e0 (table test-table [id=a490fdadc19445ce840d4a4171b1b430]) successfully deleted
I20260810 06:26:06.437861 22665 tablet_replica.cc:330] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: stopping tablet replica
I20260810 06:26:06.438148 22665 raft_consensus.cc:2229] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:26:06.438432 22665 raft_consensus.cc:2258] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:26:06.441032 22665 ts_tablet_manager.cc:1905] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:26:06.445991 22665 ts_tablet_manager.cc:1918] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:26:06.446097 22665 log.cc:1196] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/e508db2e765748388fdb5de91eaf91e0
I20260810 06:26:06.446451 22665 ts_tablet_manager.cc:1939] T e508db2e765748388fdb5de91eaf91e0 P 381acc849a7a401db33057ac3002dbdb: Deleting consensus metadata
I20260810 06:26:06.447705 21799 catalog_manager.cc:4841] TS 381acc849a7a401db33057ac3002dbdb (127.4.73.3:37885): tablet e508db2e765748388fdb5de91eaf91e0 (table test-table [id=a490fdadc19445ce840d4a4171b1b430]) successfully deleted
W20260810 06:26:06.448403 21799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:35293 (error 108)
I20260810 06:26:06.449407 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:06.449091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35293 (local address 127.0.0.1:49770)
0810 06:26:06.449229 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:06.449315 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:35293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 06:26:06.449561 21799 catalog_manager.cc:4584] TS 636097dd51e745daa53208b8531f87ce (127.4.73.1:35293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e508db2e765748388fdb5de91eaf91e0: Network error: Client connection negotiation failed: client connection to 127.4.73.1:35293: connect: Connection refused (error 111)
I20260810 06:26:06.452514 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 22170
I20260810 06:26:06.473711 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 22391
I20260810 06:26:06.491535 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 21779
2026-08-10T06:26:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2891 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:26:06 dist-test-slave-45dx krb5kdc[22674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:26:06 dist-test-slave-45dx krb5kdc[22674](info): set up 2 sockets
Aug 10 06:26:06 dist-test-slave-45dx krb5kdc[22674](info): commencing operation
krb5kdc: starting...
W20260810 06:26:07.767978 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:26:07 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:26:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/rangeradmin_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/rangeradmin_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/rangerlookup_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/rangerlookup_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
I20260810 06:26:07.935045 4388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPhbgll/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/postgres -l logfile start
2026-08-10 06:26:10.387 UTC [22724] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 06:26:10.387 UTC [22724] LOG: listening on IPv4 address "127.4.73.20", port 36101
2026-08-10 06:26:10.391 UTC [22724] LOG: listening on Unix socket "/tmp/.s.PGSQL.36101"
2026-08-10 06:26:10.419 UTC [22727] LOG: database system was shut down at 2026-08-10 06:26:09 UTC
2026-08-10 06:26:10.423 UTC [22724] LOG: database system is ready to accept connections
I20260810 06:26:11.592878 4388 mini_postgres.cc:96] Postgres bound to 36101
2026-08-10 06:26:11.602 UTC [22740] FATAL: database "slave" does not exist
127.4.73.20:36101 - accepting connections
I20260810 06:26:11.603296 4388 mini_ranger.cc:161] Starting Ranger...
I20260810 06:26:11.620190 4388 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 06:26:12.078020 4388 mini_ranger.cc:87] Created ranger Postgres database
I20260810 06:26:12.078125 4388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-admin
2026-08-10 06:26:12,484 [I] DB FLAVOR :POSTGRES
2026-08-10 06:26:12,485 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:26:12,485 [I] Checking connection..
2026-08-10 06:26:12,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 06:26:12,747 [I] Checking connection passed.
2026-08-10 06:26:12,748 [I] --------- Verifying version history table ---------
2026-08-10 06:26:12,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:26:12,985 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:26:12,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:26:13,231 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:26:13,231 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 06:26:13,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 06:26:13.494 UTC [22853] WARNING: there is no transaction in progress
2026-08-10 06:26:13,501 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 06:26:13,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:26:13,768 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 06:26:13,768 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:26:13,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 06:26:14,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 06:26:14,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-45dx.c.gcp-upstream.internal','N') ;"
2026-08-10 06:26:14,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:26:14,774 [I] Table x_portal_user does not exist in database ranger
2026-08-10 06:26:14,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:26:15,006 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 06:26:15,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:26:15,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:26:15,503 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 06:26:15,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 06:26:16.434 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.448 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.467 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.481 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.496 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.513 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.584 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.592 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.609 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.618 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.628 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.642 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.651 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.656 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.663 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:16.668 UTC [23021] WARNING: there is no transaction in progress
2026-08-10 06:26:17,264 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 06:26:17,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 06:26:17,534 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 06:26:17,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:26:17,771 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 06:26:17,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:26:18,019 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 06:26:18,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:26:18,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:26:18,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-45dx.c.gcp-upstream.internal';"
2026-08-10 06:26:18,780 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 06:26:18,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 06:26:19,036 [I] DB_PATCHES have already been applied
I20260810 06:26:19.041977 4388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPhbgll/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 06:26:19.042042 4388 mini_ranger.cc:193] Using host: 127.4.73.20
I20260810 06:26:19.046562 4388 mini_ranger.cc:227] Ranger admin URL: http://127.4.73.20:34667
Aug 10, 2026 6:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:26:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:26:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 06:26:20.445647 4388 mini_ranger.cc:160] Time spent starting Ranger: real 8.842s user 0.000s sys 0.012s
Aug 10, 2026 6:26:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:26:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 6:26:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:26:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:26:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 06:26:43.135319 4388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/rangerkms_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/rangerkms_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.73.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.73.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 06:26:43 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 06:26:43.225193 4388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 06:26:43.236151 4388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 06:26:44.049598 4388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 06:26:44.049734 4388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms
I20260810 06:26:44.049832 4388 mini_ranger_kms.cc:227] Using postgres at 127.4.73.20:36101
2026-08-10 06:26:44,473 [I] DB FLAVOR :POSTGRES
2026-08-10 06:26:44,473 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:26:44,473 [I] Checking connection
2026-08-10 06:26:44,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 06:26:44,753 [I] connection success
2026-08-10 06:26:44,754 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 06:26:44,754 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 06:26:44,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 06:26:45,023 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 06:26:45,023 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:26:45,023 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 06:26:45,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36101/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 06:26:45,301 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 06:26:45.773994 4388 mini_ranger_kms.cc:310] Created kms service
I20260810 06:26:45.972474 4388 mini_ranger_kms.cc:326] Created kudu user
I20260810 06:26:46.034034 4388 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 06:26:46 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343203, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:26:46.505100 4388 mini_ranger_kms.cc:384] Added ranger policy
I20260810 06:26:46.505307 4388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 06:26:46.505340 4388 mini_ranger_kms.cc:240] Using host: 127.4.73.20
I20260810 06:26:46.510277 4388 mini_ranger_kms.cc:276] Ranger KMS PID: 23337
I20260810 06:26:46.510443 4388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.73.20:39363
Aug 10, 2026 6:26:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:26:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:26:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:26:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:26:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:26:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 06:26:49 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786343209, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.73.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:49.339304 4388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.114s user 0.000s sys 0.398s
I20260810 06:26:49.339455 4388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 06:26:49.339583 4388 mini_ranger_kms.cc:401] 127.4.73.20:39363/kms/v1/keys
Aug 10 06:26:49 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343209, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.73.20@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
Aug 10 06:26:49 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343209, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.73.20@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
Aug 10 06:26:52 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:26:52.283677 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:40027
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:40027
--ranger_config_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:39363/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:26:52.418128 23425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:52.418416 23425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:52.418498 23425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:52.422761 23425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:26:52.422869 23425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:52.422917 23425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:52.422986 23425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:26:52.423023 23425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:26:52.428318 23425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.73.20:39363/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:40027
--ranger_config_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:40027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:52.429332 23425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:52.430341 23425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:52.439989 23433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:52.440128 23430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:52.440119 23425 server_base.cc:1034] running on GCE node
W20260810 06:26:52.440032 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:52.440965 23425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:52.442029 23425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:52.443213 23425 hybrid_clock.cc:648] HybridClock initialized: now 1786343212443194 us; error 30 us; skew 500 ppm
Aug 10 06:26:52 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343212, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:52.451490 23425 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:26:52.452514 23425 webserver.cc:460] Webserver started at http://127.4.73.62:40565/ using document root <none> and password file <none>
I20260810 06:26:52.453064 23425 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:52.453153 23425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:52.453378 23425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:26:52 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343212, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:26:52.516794 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "ed112d483c3a49dc817d7140034439bd"
format_stamp: "Formatted at 2026-08-10 06:26:52 on dist-test-slave-45dx"
server_key: "654b0c1ba3ff62010c3ecdc295b92468"
server_key_iv: "37f7c91afa6af2a48a0d1b998972e575"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:52.517730 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "ed112d483c3a49dc817d7140034439bd"
format_stamp: "Formatted at 2026-08-10 06:26:52 on dist-test-slave-45dx"
server_key: "654b0c1ba3ff62010c3ecdc295b92468"
server_key_iv: "37f7c91afa6af2a48a0d1b998972e575"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:52.523315 23425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 06:26:52.537626 23440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:52.539080 23425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 06:26:52.539361 23425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "ed112d483c3a49dc817d7140034439bd"
format_stamp: "Formatted at 2026-08-10 06:26:52 on dist-test-slave-45dx"
server_key: "654b0c1ba3ff62010c3ecdc295b92468"
server_key_iv: "37f7c91afa6af2a48a0d1b998972e575"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:52.539506 23425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:52.556682 23425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:52.559628 23425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:52.559883 23425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:52.572227 23425 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:40027
I20260810 06:26:52.572268 23526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:40027 every 8 connection(s)
I20260810 06:26:52.573561 23425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:26:52.576473 23528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:52.582389 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 23425
I20260810 06:26:52.582517 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:26:52.582520 23528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd: Bootstrap starting.
I20260810 06:26:52.582830 4388 external_mini_cluster.cc:1428] Setting key 4f61263189d5482b2614e7e8bf930e42
I20260810 06:26:52.585996 23528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:52.587070 23528 log.cc:824] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:52.589382 23528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd: No bootstrap required, opened a new log
Aug 10 06:26:52 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:52.593755 23528 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed112d483c3a49dc817d7140034439bd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40027 } }
I20260810 06:26:52.593986 23528 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:52.594061 23528 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed112d483c3a49dc817d7140034439bd, State: Initialized, Role: FOLLOWER
I20260810 06:26:52.594614 23528 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed112d483c3a49dc817d7140034439bd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40027 } }
I20260810 06:26:52.594792 23528 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:26:52.594877 23528 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:26:52.595006 23528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:52.595865 23528 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed112d483c3a49dc817d7140034439bd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40027 } }
I20260810 06:26:52.596242 23528 leader_election.cc:302] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed112d483c3a49dc817d7140034439bd; no voters:
I20260810 06:26:52.596591 23528 leader_election.cc:288] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:26:52.596673 23538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:26:52.597410 23538 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [term 1 LEADER]: Becoming Leader. State: Replica: ed112d483c3a49dc817d7140034439bd, State: Running, Role: LEADER
I20260810 06:26:52.597728 23538 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed112d483c3a49dc817d7140034439bd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40027 } }
I20260810 06:26:52.597827 23528 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:26:52.599421 23539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed112d483c3a49dc817d7140034439bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed112d483c3a49dc817d7140034439bd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40027 } } }
I20260810 06:26:52.599786 23539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [sys.catalog]: This master's current role is: LEADER
I20260810 06:26:52.599685 23540 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed112d483c3a49dc817d7140034439bd. Latest consensus state: current_term: 1 leader_uuid: "ed112d483c3a49dc817d7140034439bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed112d483c3a49dc817d7140034439bd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40027 } } }
I20260810 06:26:52.600171 23540 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd [sys.catalog]: This master's current role is: LEADER
I20260810 06:26:52.601133 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:52.584281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.4.73.62:40027)
0810 06:26:52.584931 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:52.584975 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:52.585019 (+ 44us) server_negotiation.cc:407] Connection header received
0810 06:26:52.586075 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:52.586100 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:52.586532 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:52.586888 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:52.587797 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:52.589294 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:52.590525 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:52.591272 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:52.591482 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:52.594203 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:52.594209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:52.594225 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:52.594255 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:52.596396 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:52.596960 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:52.596966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:52.596972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:52.597099 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:52.597390 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:52.597393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:52.597394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:52.597907 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:52.598593 (+ 686us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:52.598747 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 06:26:52.599065 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":226,"threads_started":1}
I20260810 06:26:52.601692 23551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:26:52.604885 23551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:26:52.605469 23528 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 06:26:52.615464 23551 catalog_manager.cc:1347] Generated new cluster ID: 7245f0b947694f64b6da97f7076f55b7
I20260810 06:26:52.615537 23551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:26:52.622368 23551 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:26:52.623387 23551 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:26:52.633219 23551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd: Generated new TSK 0
I20260810 06:26:52.633915 23551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:26:53.461436 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:39363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:40027
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:26:53.610380 23605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:53.610677 23605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:53.610790 23605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:53.614900 23605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:53.614990 23605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:53.615095 23605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:26:53.620877 23605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.73.20:39363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:53.622164 23605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:53.623418 23605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:53.635953 23614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:53.635967 23615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:53.636018 23619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:53.636847 23605 server_base.cc:1034] running on GCE node
I20260810 06:26:53.637370 23605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:53.638126 23605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:53.639339 23605 hybrid_clock.cc:648] HybridClock initialized: now 1786343213639299 us; error 46 us; skew 500 ppm
Aug 10 06:26:53 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:53.643522 23605 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:26:53.644660 23605 webserver.cc:460] Webserver started at http://127.4.73.1:37651/ using document root <none> and password file <none>
I20260810 06:26:53.645234 23605 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:53.645296 23605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:53.645526 23605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:26:53 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:26:53.661396 23605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "5171d7d6a0344cb7b7f0c198cf2505e1"
format_stamp: "Formatted at 2026-08-10 06:26:53 on dist-test-slave-45dx"
server_key: "47df2bf36c0188c8ae6e62a8686e39e4"
server_key_iv: "dc75b578dcc9437cc2a03f915ddb563e"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:53.662060 23605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "5171d7d6a0344cb7b7f0c198cf2505e1"
format_stamp: "Formatted at 2026-08-10 06:26:53 on dist-test-slave-45dx"
server_key: "47df2bf36c0188c8ae6e62a8686e39e4"
server_key_iv: "dc75b578dcc9437cc2a03f915ddb563e"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:53.666913 23605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 06:26:53.683688 23629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:53.685436 23605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:26:53.685679 23605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "5171d7d6a0344cb7b7f0c198cf2505e1"
format_stamp: "Formatted at 2026-08-10 06:26:53 on dist-test-slave-45dx"
server_key: "47df2bf36c0188c8ae6e62a8686e39e4"
server_key_iv: "dc75b578dcc9437cc2a03f915ddb563e"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:53.685902 23605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:53.714815 23605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:53.718658 23605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:53.718942 23605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:53.719725 23605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:53.720809 23605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:53.720922 23605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:53.721052 23605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:53.721131 23605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:53.738922 23605 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:35329
I20260810 06:26:53.739004 23819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:35329 every 8 connection(s)
I20260810 06:26:53.739969 23605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:26:53.750078 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 23605
I20260810 06:26:53.750231 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:26:53.750571 4388 external_mini_cluster.cc:1428] Setting key 6df501d9462ba2e284444882424413ce
Aug 10 06:26:53 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:53.757781 23824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:53.742749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:40027 (local address 127.4.73.1:38689)
0810 06:26:53.743367 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:53.743420 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:53.744498 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:53.745145 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:53.745152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:53.745433 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:53.745967 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:53.745982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:53.746838 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:53.746844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:53.747818 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:53.747832 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:53.748368 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:53.748373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:53.748582 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:53.749320 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:53.749404 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:53.751425 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:53.753887 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:53.753895 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:53.753914 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:53.754299 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:53.754704 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:53.754708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:53.754719 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:53.754992 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:53.755671 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:53.755679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:53.756022 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 06:26:53.756247 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 06:26:53.756661 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":192,"threads_started":1}
I20260810 06:26:53.758566 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:53.742409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:38689 (local address 127.4.73.62:40027)
0810 06:26:53.742728 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:53.742734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:53.743575 (+ 841us) server_negotiation.cc:407] Connection header received
0810 06:26:53.744788 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:53.744793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:53.744876 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:53.745011 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:53.746116 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:53.746708 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:53.747955 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:53.748188 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:53.748299 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:53.751608 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:53.751612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:53.751615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:53.751638 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:53.753743 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:53.754435 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:53.754441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:53.754446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:53.754545 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:53.755176 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:53.755179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:53.755181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:53.755508 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:53.755610 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:53.758201 (+ 2591us) server_negotiation.cc:299] Negotiation successful
0810 06:26:53.758346 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260810 06:26:53.759852 23820 heartbeater.cc:347] Connected to a master server at 127.4.73.62:40027
I20260810 06:26:53.760092 23820 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:53.760732 23820 heartbeater.cc:511] Master 127.4.73.62:40027 requested a full tablet report, sending...
I20260810 06:26:53.762668 23466 ts_manager.cc:194] Registered new tserver with Master: 5171d7d6a0344cb7b7f0c198cf2505e1 (127.4.73.1:35329)
I20260810 06:26:53.764420 23466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:38689
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:26:53.788219 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:39363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:40027
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:26:53 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786343213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 06:26:53.975651 23827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:53.976073 23827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:53.976174 23827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:53.982231 23827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:53.982376 23827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:53.982542 23827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:26:53.990530 23827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.73.20:39363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:53.992034 23827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:53.993245 23827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:26:54.005159 23840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:54.005273 23844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:54.005734 23842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:54.006805 23827 server_base.cc:1034] running on GCE node
I20260810 06:26:54.007288 23827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:54.008068 23827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:54.009269 23827 hybrid_clock.cc:648] HybridClock initialized: now 1786343214009212 us; error 72 us; skew 500 ppm
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:54.013387 23827 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:26:54.014667 23827 webserver.cc:460] Webserver started at http://127.4.73.2:33081/ using document root <none> and password file <none>
I20260810 06:26:54.015412 23827 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:54.015513 23827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:54.015776 23827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:26:54.033578 23827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "40de7027f0e4451cae42cb683971e14e"
format_stamp: "Formatted at 2026-08-10 06:26:54 on dist-test-slave-45dx"
server_key: "07d0b0e26d1c66d90a70dcdd142dddd0"
server_key_iv: "4365483d4c3356f61d3444f8d7b91eae"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:54.034452 23827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "40de7027f0e4451cae42cb683971e14e"
format_stamp: "Formatted at 2026-08-10 06:26:54 on dist-test-slave-45dx"
server_key: "07d0b0e26d1c66d90a70dcdd142dddd0"
server_key_iv: "4365483d4c3356f61d3444f8d7b91eae"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:54.040364 23827 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 06:26:54.074142 23857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:54.075951 23827 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.001s
I20260810 06:26:54.076185 23827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "40de7027f0e4451cae42cb683971e14e"
format_stamp: "Formatted at 2026-08-10 06:26:54 on dist-test-slave-45dx"
server_key: "07d0b0e26d1c66d90a70dcdd142dddd0"
server_key_iv: "4365483d4c3356f61d3444f8d7b91eae"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:54.076439 23827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:54.114253 23827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:54.118029 23827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:54.118278 23827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:54.119130 23827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:54.120415 23827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:54.120468 23827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:54.120512 23827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:54.120558 23827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:54.137598 23827 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:45547
I20260810 06:26:54.138914 23827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:26:54.141003 24055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:45547 every 8 connection(s)
I20260810 06:26:54.148931 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 23827
I20260810 06:26:54.149057 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:26:54.149428 4388 external_mini_cluster.cc:1428] Setting key 2dfa9ac847364cf3205af6f73e07f7fa
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:54.161473 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.141630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:58975 (local address 127.4.73.62:40027)
0810 06:26:54.141801 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.141806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.142846 (+ 1040us) server_negotiation.cc:407] Connection header received
0810 06:26:54.144149 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.144154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.144215 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.144381 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:54.146000 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.146943 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.148823 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.149166 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.149252 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.156482 (+ 7230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:54.156488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:54.156493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:54.156529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:54.158567 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:54.159244 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:54.159249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:54.159252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:54.159346 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:54.159820 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:54.159826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:54.159829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:54.160083 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:54.160166 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.161035 (+ 869us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.161180 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:26:54.163594 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.142087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:40027 (local address 127.4.73.2:58975)
0810 06:26:54.142642 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.142697 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.143939 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.144596 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:54.144605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:54.145034 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:54.145799 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.145822 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.147152 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.147155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.148678 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.148689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.149544 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.149547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.149744 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.150463 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:54.150484 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:54.152662 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:54.158726 (+ 6064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:54.158736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:54.158754 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:54.159117 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:54.159489 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:54.159491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:54.159494 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:54.159698 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:54.160257 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:54.160265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:54.160658 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 06:26:54.162019 (+ 1361us) client_negotiation.cc:240] Negotiation successful
0810 06:26:54.162443 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:26:54.165234 24056 heartbeater.cc:347] Connected to a master server at 127.4.73.62:40027
I20260810 06:26:54.165480 24056 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:54.165978 24056 heartbeater.cc:511] Master 127.4.73.62:40027 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:26:54.167171 23466 ts_manager.cc:194] Registered new tserver with Master: 40de7027f0e4451cae42cb683971e14e (127.4.73.2:45547)
I20260810 06:26:54.167980 23466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:58975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:26:54.185402 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:39363/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:40027
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:26:54.386245 24067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:26:54.386797 24067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:26:54.386933 24067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:26:54.393651 24067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:26:54.393743 24067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:26:54.393857 24067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:26:54.402878 24067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.73.20:39363/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:40027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:26:54.404603 24067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:26:54.406320 24067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
W20260810 06:26:54.430577 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:54.430994 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:26:54.431540 24075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:26:54.436581 24067 server_base.cc:1034] running on GCE node
I20260810 06:26:54.437275 24067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:26:54.438112 24067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:26:54.439309 24067 hybrid_clock.cc:648] HybridClock initialized: now 1786343214439278 us; error 39 us; skew 500 ppm
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:26:54.443785 24067 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:26:54.444865 24067 webserver.cc:460] Webserver started at http://127.4.73.3:42911/ using document root <none> and password file <none>
I20260810 06:26:54.445590 24067 fs_manager.cc:359] Metadata directory not provided
I20260810 06:26:54.445713 24067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:26:54.445986 24067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:26:54.480089 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f"
format_stamp: "Formatted at 2026-08-10 06:26:54 on dist-test-slave-45dx"
server_key: "de74865d59414470a3c7cc5b654ec747"
server_key_iv: "5e7ab26ecda273f936e30c09f42c948d"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:54.480926 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f"
format_stamp: "Formatted at 2026-08-10 06:26:54 on dist-test-slave-45dx"
server_key: "de74865d59414470a3c7cc5b654ec747"
server_key_iv: "5e7ab26ecda273f936e30c09f42c948d"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:54.492934 24067 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260810 06:26:54.528942 24097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:54.530679 24067 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.004s sys 0.000s
I20260810 06:26:54.530869 24067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f"
format_stamp: "Formatted at 2026-08-10 06:26:54 on dist-test-slave-45dx"
server_key: "de74865d59414470a3c7cc5b654ec747"
server_key_iv: "5e7ab26ecda273f936e30c09f42c948d"
server_key_version: "kuduclusterkey@0"
I20260810 06:26:54.531005 24067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:26:54.553726 24067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:26:54.557415 24067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:26:54.557648 24067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:26:54.558382 24067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:26:54.559512 24067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:26:54.559628 24067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:54.559727 24067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:26:54.559803 24067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:26:54.578238 24067 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:37731
I20260810 06:26:54.578337 24295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:37731 every 8 connection(s)
I20260810 06:26:54.579486 24067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:26:54.581178 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 24067
I20260810 06:26:54.581310 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:26:54.581630 4388 external_mini_cluster.cc:1428] Setting key f45eac77736b6e5a89ede6714f64ed6d
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343214, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:26:54.597532 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.581704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:36309 (local address 127.4.73.62:40027)
0810 06:26:54.581861 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.581866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.583237 (+ 1371us) server_negotiation.cc:407] Connection header received
0810 06:26:54.584485 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.584488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.584538 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.584604 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:54.586032 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.586548 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.589340 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.589580 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.589670 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.593274 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:54.593277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:54.593278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:54.593304 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:54.595233 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:54.595875 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:54.595878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:54.595879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:54.595984 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:54.596379 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:54.596382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:54.596383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:54.596571 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:54.596678 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.597247 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.597381 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:26:54.598522 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.582283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:40027 (local address 127.4.73.3:36309)
0810 06:26:54.582960 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.583140 (+ 180us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.584204 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.584777 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:54.584786 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:54.585173 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:26:54.585856 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.585875 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.586700 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.586712 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.589085 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.589101 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.589830 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.589837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.590160 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.591170 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:26:54.591209 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:26:54.593108 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:26:54.595357 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:54.595363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:54.595376 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:54.595759 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:54.596101 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:26:54.596104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:26:54.596106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:26:54.596252 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:26:54.596677 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:26:54.596683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:26:54.596976 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 06:26:54.597174 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 06:26:54.597494 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":235,"threads_started":1}
I20260810 06:26:54.599745 24296 heartbeater.cc:347] Connected to a master server at 127.4.73.62:40027
I20260810 06:26:54.600099 24296 heartbeater.cc:464] Registering TS with master...
I20260810 06:26:54.600865 24296 heartbeater.cc:511] Master 127.4.73.62:40027 requested a full tablet report, sending...
I20260810 06:26:54.601909 23466 ts_manager.cc:194] Registered new tserver with Master: ca6d2c9ddfb84d7e83d848948c2ca59f (127.4.73.3:37731)
I20260810 06:26:54.602717 23466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:36309
I20260810 06:26:54.610948 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:26:54.624002 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.613623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.4.73.62:40027)
0810 06:26:54.613761 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.613765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.614134 (+ 369us) server_negotiation.cc:407] Connection header received
0810 06:26:54.614245 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.614248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.614308 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.614399 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:26:54.615296 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.616090 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.617353 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.617712 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.617846 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.619466 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:26:54.619470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:26:54.619473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:26:54.619500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:26:54.622021 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:54.622551 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:54.622555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:54.622557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:54.622667 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:26:54.623036 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:26:54.623040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:26:54.623041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:26:54.623281 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:26:54.623424 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.623683 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.623823 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 06:26:54.627300 23466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:26:54.632043 23466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:26:54.652189 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.643306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:45547 (local address 127.0.0.1:47998)
0810 06:26:54.644594 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.645219 (+ 625us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.645495 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.646993 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:54.646999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:54.647022 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:54.647169 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.647175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.648830 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.648834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.650179 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.650190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.651605 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.651608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.651717 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.651766 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 06:26:54.651912 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 06:26:54.652035 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"thread_start_us":208,"threads_started":1}
I20260810 06:26:54.652953 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.643485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.4.73.2:45547)
0810 06:26:54.644223 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.644275 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.645317 (+ 1042us) server_negotiation.cc:407] Connection header received
0810 06:26:54.646410 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.646416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.646635 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.646798 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:54.647306 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.648686 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.650358 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.651268 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.651383 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.651537 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.652641 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.652760 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":183,"threads_started":1}
I20260810 06:26:54.656486 23946 tablet_service.cc:1467] Processing CreateTablet for tablet 099e6203a3c3478a9225e8722a47bbf9 (DEFAULT_TABLE table=test-table [id=26a1aca46a034f31a803c5e2186edcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:26:54.657569 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.645591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:37731 (local address 127.0.0.1:44278)
0810 06:26:54.647717 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.647748 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.647876 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.649297 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:54.649300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:54.649316 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:54.649447 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.649453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.650993 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.650996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.655009 (+ 4013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.655017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.657068 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.657072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.657271 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.657304 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 06:26:54.657376 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 06:26:54.657441 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2068,"thread_start_us":119,"threads_started":1}
I20260810 06:26:54.657938 23946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099e6203a3c3478a9225e8722a47bbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:54.661716 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.645722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.4.73.3:37731)
0810 06:26:54.647512 (+ 1790us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.647544 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.647965 (+ 421us) server_negotiation.cc:407] Connection header received
0810 06:26:54.648043 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.648047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.648213 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.649146 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:54.649602 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.650843 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.655205 (+ 4362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.656754 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.656876 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.657007 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.661403 (+ 4396us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.661504 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1687,"thread_start_us":182,"threads_started":1}
I20260810 06:26:54.664891 24182 tablet_service.cc:1467] Processing CreateTablet for tablet 099e6203a3c3478a9225e8722a47bbf9 (DEFAULT_TABLE table=test-table [id=26a1aca46a034f31a803c5e2186edcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:26:54.665942 24182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099e6203a3c3478a9225e8722a47bbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:54.671140 24330 tablet_bootstrap.cc:492] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Bootstrap starting.
I20260810 06:26:54.673892 24330 tablet_bootstrap.cc:654] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:54.675254 24330 log.cc:824] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:54.676602 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.644680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35329 (local address 127.0.0.1:37976)
0810 06:26:54.646126 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.646140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.646227 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.654559 (+ 8332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:54.654562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:54.654578 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:54.654671 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.654676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.659345 (+ 4669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.659348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.661171 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.661181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.676134 (+ 14953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.676138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.676252 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.676277 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:26:54.676381 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 06:26:54.676463 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"thread_start_us":157,"threads_started":1}
I20260810 06:26:54.677673 24330 tablet_bootstrap.cc:492] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: No bootstrap required, opened a new log
I20260810 06:26:54.677886 24330 ts_tablet_manager.cc:1397] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20260810 06:26:54.679682 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.644821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.4.73.1:35329)
0810 06:26:54.654015 (+ 9194us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.654021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.654043 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:26:54.654113 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.654118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.654281 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.654427 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:54.657838 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.659203 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.674148 (+ 14945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.675822 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.675933 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.676070 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.679415 (+ 3345us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.679538 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9092,"thread_start_us":219,"threads_started":1}
I20260810 06:26:54.682717 24330 raft_consensus.cc:348] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.682988 24330 raft_consensus.cc:374] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:54.682578 23710 tablet_service.cc:1467] Processing CreateTablet for tablet 099e6203a3c3478a9225e8722a47bbf9 (DEFAULT_TABLE table=test-table [id=26a1aca46a034f31a803c5e2186edcfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:26:54.683041 24330 raft_consensus.cc:729] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40de7027f0e4451cae42cb683971e14e, State: Initialized, Role: FOLLOWER
I20260810 06:26:54.683555 24330 consensus_queue.cc:260] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.683704 23710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 099e6203a3c3478a9225e8722a47bbf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:26:54.685173 24330 ts_tablet_manager.cc:1428] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260810 06:26:54.685632 24056 heartbeater.cc:503] Master 127.4.73.62:40027 was elected leader, sending a full tablet report...
I20260810 06:26:54.691973 24332 tablet_bootstrap.cc:492] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Bootstrap starting.
I20260810 06:26:54.694314 24337 tablet_bootstrap.cc:492] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1: Bootstrap starting.
I20260810 06:26:54.695181 24332 tablet_bootstrap.cc:654] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:54.696144 24332 log.cc:824] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:54.697613 24337 tablet_bootstrap.cc:654] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1: Neither blocks nor log segments found. Creating new log.
I20260810 06:26:54.698482 24332 tablet_bootstrap.cc:492] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: No bootstrap required, opened a new log
I20260810 06:26:54.698683 24337 log.cc:824] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1: Log is configured to *not* fsync() on all Append() calls
I20260810 06:26:54.698760 24332 ts_tablet_manager.cc:1397] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 06:26:54.700825 24337 tablet_bootstrap.cc:492] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1: No bootstrap required, opened a new log
I20260810 06:26:54.701022 24337 ts_tablet_manager.cc:1397] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 06:26:54.704219 24332 raft_consensus.cc:348] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.704555 24332 raft_consensus.cc:374] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:54.704619 24332 raft_consensus.cc:729] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6d2c9ddfb84d7e83d848948c2ca59f, State: Initialized, Role: FOLLOWER
I20260810 06:26:54.705085 24332 consensus_queue.cc:260] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.706175 24332 ts_tablet_manager.cc:1428] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260810 06:26:54.706565 24296 heartbeater.cc:503] Master 127.4.73.62:40027 was elected leader, sending a full tablet report...
I20260810 06:26:54.709573 24337 raft_consensus.cc:348] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.709846 24337 raft_consensus.cc:374] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:26:54.709896 24337 raft_consensus.cc:729] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5171d7d6a0344cb7b7f0c198cf2505e1, State: Initialized, Role: FOLLOWER
I20260810 06:26:54.710354 24337 consensus_queue.cc:260] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.711099 23820 heartbeater.cc:503] Master 127.4.73.62:40027 was elected leader, sending a full tablet report...
I20260810 06:26:54.711557 24337 ts_tablet_manager.cc:1428] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260810 06:26:54.732227 24336 raft_consensus.cc:482] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:26:54.732515 24336 raft_consensus.cc:504] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.733934 24336 leader_election.cc:288] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5171d7d6a0344cb7b7f0c198cf2505e1 (127.4.73.1:35329), ca6d2c9ddfb84d7e83d848948c2ca59f (127.4.73.3:37731)
I20260810 06:26:54.739504 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.734919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:41951 (local address 127.4.73.1:35329)
0810 06:26:54.735033 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.735037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.735072 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:26:54.735127 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.735130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.735179 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.735249 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:54.735685 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.736499 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.738184 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.738800 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.738877 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.738952 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.739303 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.739379 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:26:54.739933 24346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.734195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35329 (local address 127.4.73.2:41951)
0810 06:26:54.734851 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.734868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.734997 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.735383 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:54.735386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:54.735406 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:54.735552 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.735558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.736715 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.736718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.738021 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.738033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.739058 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.739061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.739163 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.739185 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:26:54.739706 (+ 521us) client_negotiation.cc:240] Negotiation successful
0810 06:26:54.739787 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":177,"threads_started":1}
I20260810 06:26:54.740643 23736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e6203a3c3478a9225e8722a47bbf9" candidate_uuid: "40de7027f0e4451cae42cb683971e14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" is_pre_election: true
I20260810 06:26:54.740993 23736 raft_consensus.cc:2454] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40de7027f0e4451cae42cb683971e14e in term 0.
I20260810 06:26:54.741532 23859 leader_election.cc:302] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40de7027f0e4451cae42cb683971e14e, 5171d7d6a0344cb7b7f0c198cf2505e1; no voters:
W20260810 06:26:54.742002 23822 tablet.cc:2369] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:26:54.742743 24336 raft_consensus.cc:2781] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:26:54.742841 24336 raft_consensus.cc:482] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:26:54.742893 24336 raft_consensus.cc:3037] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:54.744323 24336 raft_consensus.cc:504] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.744688 24336 leader_election.cc:288] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [CANDIDATE]: Term 1 election: Requested vote from peers 5171d7d6a0344cb7b7f0c198cf2505e1 (127.4.73.1:35329), ca6d2c9ddfb84d7e83d848948c2ca59f (127.4.73.3:37731)
I20260810 06:26:54.745154 23736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e6203a3c3478a9225e8722a47bbf9" candidate_uuid: "40de7027f0e4451cae42cb683971e14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1"
I20260810 06:26:54.745278 23736 raft_consensus.cc:3037] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:54.745721 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.734679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:33485 (local address 127.4.73.3:37731)
0810 06:26:54.734812 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.734816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.735419 (+ 603us) server_negotiation.cc:407] Connection header received
0810 06:26:54.735640 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.735645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.735693 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.735739 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:26:54.736258 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.736989 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.741804 (+ 4815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.742377 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.742464 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.742525 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.745435 (+ 2910us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.745576 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:26:54.746271 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.734268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:37731 (local address 127.4.73.2:33485)
0810 06:26:54.735291 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.735305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.735499 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.735923 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:26:54.735926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:26:54.735943 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:26:54.736112 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.736122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.739945 (+ 3823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.739948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.741558 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:26:54.741570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.745045 (+ 3475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.745053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:26:54.745193 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.745217 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:26:54.746006 (+ 789us) client_negotiation.cc:240] Negotiation successful
0810 06:26:54.746080 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":121,"threads_started":1}
I20260810 06:26:54.746641 23736 raft_consensus.cc:2454] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40de7027f0e4451cae42cb683971e14e in term 1.
I20260810 06:26:54.746964 23859 leader_election.cc:302] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40de7027f0e4451cae42cb683971e14e, 5171d7d6a0344cb7b7f0c198cf2505e1; no voters:
I20260810 06:26:54.747005 24213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e6203a3c3478a9225e8722a47bbf9" candidate_uuid: "40de7027f0e4451cae42cb683971e14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" is_pre_election: true
I20260810 06:26:54.747123 24336 raft_consensus.cc:2781] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:26:54.747334 24336 raft_consensus.cc:686] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 1 LEADER]: Becoming Leader. State: Replica: 40de7027f0e4451cae42cb683971e14e, State: Running, Role: LEADER
I20260810 06:26:54.747361 24212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "099e6203a3c3478a9225e8722a47bbf9" candidate_uuid: "40de7027f0e4451cae42cb683971e14e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f"
I20260810 06:26:54.747684 24336 consensus_queue.cc:237] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } }
I20260810 06:26:54.747367 24213 raft_consensus.cc:2454] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40de7027f0e4451cae42cb683971e14e in term 0.
I20260810 06:26:54.750789 23465 catalog_manager.cc:5495] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e reported cstate change: term changed from 0 to 1, leader changed from <none> to 40de7027f0e4451cae42cb683971e14e (127.4.73.2). New cstate: current_term: 1 leader_uuid: "40de7027f0e4451cae42cb683971e14e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40de7027f0e4451cae42cb683971e14e" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 45547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 } health_report { overall_health: UNKNOWN } } }
I20260810 06:26:54.778756 24322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:26:54.773673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.4.73.2:45547)
0810 06:26:54.773812 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:26:54.773815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:26:54.773842 (+ 27us) server_negotiation.cc:407] Connection header received
0810 06:26:54.773915 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:26:54.773917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:26:54.773972 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:26:54.774101 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:26:54.774374 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.775189 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.776494 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:26:54.776842 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:26:54.776973 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:26:54.777391 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:26:54.778013 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:26:54.778133 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 06:26:54.778542 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 06:26:54.778609 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:26:54.786260 24213 raft_consensus.cc:3037] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:26:54.788192 24213 raft_consensus.cc:1261] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 1 FOLLOWER]: Refusing update from remote peer 40de7027f0e4451cae42cb683971e14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:26:54.789042 24349 consensus_queue.cc:1035] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca6d2c9ddfb84d7e83d848948c2ca59f" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:26:54.789489 23736 raft_consensus.cc:1261] T 099e6203a3c3478a9225e8722a47bbf9 P 5171d7d6a0344cb7b7f0c198cf2505e1 [term 1 FOLLOWER]: Refusing update from remote peer 40de7027f0e4451cae42cb683971e14e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:26:54.790187 24349 consensus_queue.cc:1035] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5171d7d6a0344cb7b7f0c198cf2505e1" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Aug 10 06:26:54 dist-test-slave-45dx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343213, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:26:54.804595 24354 mvcc.cc:205] Tried to move back new op lower bound from 7316861807755378688 to 7316861807610064896. Current Snapshot: MvccSnapshot[applied={T|T < 7316861807755378688}]
I20260810 06:26:54.811197 24356 mvcc.cc:205] Tried to move back new op lower bound from 7316861807755378688 to 7316861807610064896. Current Snapshot: MvccSnapshot[applied={T|T < 7316861807755378688}]
I20260810 06:26:54.816084 24359 mvcc.cc:205] Tried to move back new op lower bound from 7316861807755378688 to 7316861807610064896. Current Snapshot: MvccSnapshot[applied={T|T < 7316861807755378688 or (T in {7316861807755378688})}]
I20260810 06:26:54.822345 23465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:26:54.822516 23465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:26:54.825197 23465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed112d483c3a49dc817d7140034439bd: Sending DeleteTablet for 3 replicas of tablet 099e6203a3c3478a9225e8722a47bbf9
I20260810 06:26:54.826011 23946 tablet_service.cc:1514] Processing DeleteTablet for tablet 099e6203a3c3478a9225e8722a47bbf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:26:54 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:47998
I20260810 06:26:54.826087 23710 tablet_service.cc:1514] Processing DeleteTablet for tablet 099e6203a3c3478a9225e8722a47bbf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:26:54 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:37976
I20260810 06:26:54.826895 24182 tablet_service.cc:1514] Processing DeleteTablet for tablet 099e6203a3c3478a9225e8722a47bbf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:26:54 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:44278
I20260810 06:26:54.827414 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 23605
I20260810 06:26:54.843989 24374 tablet_replica.cc:330] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: stopping tablet replica
I20260810 06:26:54.844426 24374 raft_consensus.cc:2229] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:26:54.844837 24374 raft_consensus.cc:2258] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:26:54.846561 24374 ts_tablet_manager.cc:1905] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 06:26:54.854585 23443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:35329 (error 108)
I20260810 06:26:54.855566 24375 tablet_replica.cc:330] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: stopping tablet replica
I20260810 06:26:54.855942 24375 raft_consensus.cc:2229] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:26:54.856340 24375 raft_consensus.cc:2258] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:26:54.857960 24375 ts_tablet_manager.cc:1905] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:26:54.858459 24374 ts_tablet_manager.cc:1918] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:26:54.858577 24374 log.cc:1196] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/099e6203a3c3478a9225e8722a47bbf9
I20260810 06:26:54.858990 24374 ts_tablet_manager.cc:1939] T 099e6203a3c3478a9225e8722a47bbf9 P 40de7027f0e4451cae42cb683971e14e: Deleting consensus metadata
I20260810 06:26:54.860241 23445 catalog_manager.cc:4841] TS 40de7027f0e4451cae42cb683971e14e (127.4.73.2:45547): tablet 099e6203a3c3478a9225e8722a47bbf9 (table test-table [id=26a1aca46a034f31a803c5e2186edcfc]) successfully deleted
W20260810 06:26:54.860747 24327 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:26:54.855375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:35329 (local address 127.0.0.1:37978)
0810 06:26:54.855515 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 06:26:54.855533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:26:54.855641 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:26:54.860608 (+ 4967us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:35329: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 06:26:54.860987 24375 ts_tablet_manager.cc:1918] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260810 06:26:54.860991 23443 catalog_manager.cc:4584] TS 5171d7d6a0344cb7b7f0c198cf2505e1 (127.4.73.1:35329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 099e6203a3c3478a9225e8722a47bbf9: Network error: Client connection negotiation failed: client connection to 127.4.73.1:35329: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:26:54.861079 24375 log.cc:1196] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/099e6203a3c3478a9225e8722a47bbf9
I20260810 06:26:54.861394 24375 ts_tablet_manager.cc:1939] T 099e6203a3c3478a9225e8722a47bbf9 P ca6d2c9ddfb84d7e83d848948c2ca59f: Deleting consensus metadata
I20260810 06:26:54.861953 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 23827
I20260810 06:26:54.870005 23442 catalog_manager.cc:4841] TS ca6d2c9ddfb84d7e83d848948c2ca59f (127.4.73.3:37731): tablet 099e6203a3c3478a9225e8722a47bbf9 (table test-table [id=26a1aca46a034f31a803c5e2186edcfc]) successfully deleted
I20260810 06:26:54.877475 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 24067
I20260810 06:26:54.889914 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 23425
I20260810 06:26:54.918938 4388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 06:26:55.274243 4388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 06:26:55.274391 4388 mini_ranger.cc:66] Stopping Ranger...
I20260810 06:26:55.630378 4388 mini_ranger.cc:68] Stopped Ranger
2026-08-10 06:26:55.630 UTC [22724] LOG: received smart shutdown request
2026-08-10 06:26:55.634 UTC [22724] LOG: background worker "logical replication launcher" (PID 22736) exited with exit code 1
2026-08-10 06:26:55.635 UTC [22729] LOG: shutting down
2026-08-10 06:26:55.789 UTC [22724] LOG: database system is shut down
2026-08-10T06:26:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49409 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:26:55 dist-test-slave-45dx krb5kdc[24394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:26:55 dist-test-slave-45dx krb5kdc[24394](info): set up 2 sockets
Aug 10 06:26:55 dist-test-slave-45dx krb5kdc[24394](info): commencing operation
krb5kdc: starting...
W20260810 06:26:57.187193 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:26:57 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:26:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/rangeradmin_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/rangeradmin_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/rangerlookup_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/rangerlookup_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
I20260810 06:26:57.359418 4388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPhbgll/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/postgres -l logfile start
2026-08-10 06:26:59.913 UTC [24454] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 06:26:59.913 UTC [24454] LOG: listening on IPv4 address "127.4.73.20", port 36001
2026-08-10 06:26:59.917 UTC [24454] LOG: listening on Unix socket "/tmp/.s.PGSQL.36001"
2026-08-10 06:26:59.939 UTC [24459] LOG: database system was shut down at 2026-08-10 06:26:58 UTC
2026-08-10 06:26:59.943 UTC [24454] LOG: database system is ready to accept connections
I20260810 06:27:01.440069 4388 mini_postgres.cc:96] Postgres bound to 36001
2026-08-10 06:27:01.456 UTC [24476] FATAL: database "slave" does not exist
127.4.73.20:36001 - accepting connections
I20260810 06:27:01.457446 4388 mini_ranger.cc:161] Starting Ranger...
I20260810 06:27:01.494016 4388 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 06:27:02.105470 4388 mini_ranger.cc:87] Created ranger Postgres database
I20260810 06:27:02.105662 4388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-admin
2026-08-10 06:27:02,578 [I] DB FLAVOR :POSTGRES
2026-08-10 06:27:02,579 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:27:02,579 [I] Checking connection..
2026-08-10 06:27:02,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 06:27:03,001 [I] Checking connection passed.
2026-08-10 06:27:03,001 [I] --------- Verifying version history table ---------
2026-08-10 06:27:03,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:27:03,383 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:27:03,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:27:03,644 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 06:27:03,644 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 06:27:03,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 06:27:03.899 UTC [24583] WARNING: there is no transaction in progress
2026-08-10 06:27:03,908 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 06:27:03,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 06:27:04,177 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 06:27:04,177 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:27:04,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 06:27:04,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 06:27:04,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-45dx.c.gcp-upstream.internal','N') ;"
2026-08-10 06:27:04,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:27:05,218 [I] Table x_portal_user does not exist in database ranger
2026-08-10 06:27:05,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:27:05,466 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 06:27:05,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:27:05,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:27:05,983 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 06:27:05,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 06:27:06.927 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:06.945 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:06.961 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:06.976 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:06.995 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.016 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.082 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.089 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.102 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.112 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.122 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.135 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.144 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.150 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.157 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07.163 UTC [24753] WARNING: there is no transaction in progress
2026-08-10 06:27:07,789 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 06:27:07,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 06:27:08,039 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 06:27:08,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 06:27:08,295 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 06:27:08,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 06:27:08,558 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 06:27:08,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 06:27:08,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 06:27:09,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-45dx.c.gcp-upstream.internal';"
2026-08-10 06:27:09,339 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 06:27:09,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 06:27:09,583 [I] DB_PATCHES have already been applied
I20260810 06:27:09.589030 4388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPhbgll/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 06:27:09.589148 4388 mini_ranger.cc:193] Using host: 127.4.73.20
I20260810 06:27:09.593537 4388 mini_ranger.cc:227] Ranger admin URL: http://127.4.73.20:48643
Aug 10, 2026 6:27:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:27:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:27:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:27:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:27:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:27:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:27:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 06:27:10.987866 4388 mini_ranger.cc:160] Time spent starting Ranger: real 9.530s user 0.000s sys 0.018s
Aug 10, 2026 6:27:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 6:27:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 6:27:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 6:27:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 6:27:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 06:27:31.019618 4388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.73.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.73.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/rangerkms_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.73.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/rangerkms_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.73.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.73.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.73.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/HTTP_127.4.73.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 06:27:31 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343251, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 06:27:31.101773 4388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 06:27:31.111553 4388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 06:27:31.898233 4388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 06:27:31.898341 4388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms
I20260810 06:27:31.898361 4388 mini_ranger_kms.cc:227] Using postgres at 127.4.73.20:36001
2026-08-10 06:27:32,079 [I] DB FLAVOR :POSTGRES
2026-08-10 06:27:32,080 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 06:27:32,080 [I] Checking connection
2026-08-10 06:27:32,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 06:27:32,333 [I] connection success
2026-08-10 06:27:32,333 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 06:27:32,333 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 06:27:32,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 06:27:32,603 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 06:27:32,603 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 06:27:32,603 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 06:27:32,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.73.20:36001/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 06:27:32,866 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 06:27:33.329708 4388 mini_ranger_kms.cc:310] Created kms service
I20260810 06:27:33.517926 4388 mini_ranger_kms.cc:326] Created kudu user
I20260810 06:27:33.562062 4388 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 06:27:33 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343251, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:33.946482 4388 mini_ranger_kms.cc:384] Added ranger policy
I20260810 06:27:33.946713 4388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPhbgll/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPhbgll/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 06:27:33.946741 4388 mini_ranger_kms.cc:240] Using host: 127.4.73.20
I20260810 06:27:33.951512 4388 mini_ranger_kms.cc:276] Ranger KMS PID: 25076
I20260810 06:27:33.951596 4388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.73.20:57795
Aug 10, 2026 6:27:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:27:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:27:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 6:27:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 6:27:36 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 6:27:36 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 06:27:36 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786343256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.73.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:36.857014 4388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.755s user 0.000s sys 0.166s
I20260810 06:27:36.857144 4388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 06:27:36.857234 4388 mini_ranger_kms.cc:401] 127.4.73.20:57795/kms/v1/keys
Aug 10 06:27:37 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.73.20@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
Aug 10 06:27:37 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343256, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.73.20@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
Aug 10 06:27:39 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:27:39.877494 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:45289
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:45289
--ranger_config_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:39.989125 25172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:39.989404 25172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:39.989473 25172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:39.993503 25172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:39.993587 25172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:39.993625 25172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:39.993680 25172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:39.993718 25172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:39.998960 25172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.73.20:57795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:45289
--ranger_config_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:39.999967 25172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:40.001042 25172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:40.009639 25180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:40.009701 25182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:40.009809 25186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:40.009974 25172 server_base.cc:1034] running on GCE node
I20260810 06:27:40.010564 25172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:40.011621 25172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:40.012809 25172 hybrid_clock.cc:648] HybridClock initialized: now 1786343260012790 us; error 34 us; skew 500 ppm
Aug 10 06:27:40 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343260, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:40.015547 25172 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:27:40.016584 25172 webserver.cc:460] Webserver started at http://127.4.73.62:39023/ using document root <none> and password file <none>
I20260810 06:27:40.017107 25172 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:40.017169 25172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:40.017380 25172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:27:40 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343260, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:40.065428 25172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "d28ed21d029a4a459b3e39ce54d58c0c"
format_stamp: "Formatted at 2026-08-10 06:27:40 on dist-test-slave-45dx"
server_key: "05d62d475e5893d86b1ce2a8227a27c0"
server_key_iv: "2f49cf8b507f129a6572dabfff649ca2"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:40.066114 25172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "d28ed21d029a4a459b3e39ce54d58c0c"
format_stamp: "Formatted at 2026-08-10 06:27:40 on dist-test-slave-45dx"
server_key: "05d62d475e5893d86b1ce2a8227a27c0"
server_key_iv: "2f49cf8b507f129a6572dabfff649ca2"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:40.070464 25172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 06:27:40.084550 25202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:40.085907 25172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:27:40.086064 25172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "d28ed21d029a4a459b3e39ce54d58c0c"
format_stamp: "Formatted at 2026-08-10 06:27:40 on dist-test-slave-45dx"
server_key: "05d62d475e5893d86b1ce2a8227a27c0"
server_key_iv: "2f49cf8b507f129a6572dabfff649ca2"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:40.086222 25172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:40.097973 25172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:40.101080 25172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:40.101609 25172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:40.109651 25172 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:45289
I20260810 06:27:40.109655 25286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:45289 every 8 connection(s)
I20260810 06:27:40.110715 25172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:27:40.113729 25287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:40.115051 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 25172
I20260810 06:27:40.115187 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:27:40.115510 4388 external_mini_cluster.cc:1428] Setting key 2ffc076d7472b9f24136c88208500dea
I20260810 06:27:40.121477 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Bootstrap starting.
Aug 10 06:27:40 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:40.124262 25287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:40.125202 25287 log.cc:824] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:40.127590 25287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: No bootstrap required, opened a new log
I20260810 06:27:40.130453 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:40.116927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.4.73.62:45289)
0810 06:27:40.117364 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:40.117380 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:40.117416 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:27:40.118053 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:40.118069 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:40.118406 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:40.118786 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:40.119577 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:40.120645 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:40.121516 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:40.122090 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:40.122267 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:40.124928 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:40.124934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:40.124947 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:40.124975 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:40.127453 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:40.127968 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:40.127973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:40.127977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:40.128078 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:40.128533 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:40.128536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:40.128538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:40.128938 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:40.129157 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:40.129457 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 06:27:40.129757 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":143,"threads_started":1}
I20260810 06:27:40.131458 25287 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:40.131729 25287 raft_consensus.cc:374] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:40.131841 25287 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28ed21d029a4a459b3e39ce54d58c0c, State: Initialized, Role: FOLLOWER
I20260810 06:27:40.132378 25287 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:40.132534 25287 raft_consensus.cc:388] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:27:40.132637 25287 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:27:40.132797 25287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:40.133890 25287 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:40.134270 25287 leader_election.cc:302] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28ed21d029a4a459b3e39ce54d58c0c; no voters:
I20260810 06:27:40.134618 25287 leader_election.cc:288] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:27:40.134742 25297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:27:40.135057 25297 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 LEADER]: Becoming Leader. State: Replica: d28ed21d029a4a459b3e39ce54d58c0c, State: Running, Role: LEADER
I20260810 06:27:40.135397 25297 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:40.135942 25287 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:40.136554 25297 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28ed21d029a4a459b3e39ce54d58c0c. Latest consensus state: current_term: 1 leader_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } } }
I20260810 06:27:40.136780 25297 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:40.137019 25299 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } } }
I20260810 06:27:40.137166 25299 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:40.137328 25318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:27:40.140815 25318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:27:40.141345 25287 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 06:27:40.151201 25318 catalog_manager.cc:1347] Generated new cluster ID: 50803b5463bd4e669be31be582f99b6e
I20260810 06:27:40.151284 25318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:27:40.176563 25318 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:27:40.178773 25318 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:27:40.185817 25318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Generated new TSK 0
I20260810 06:27:40.187568 25318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:27:40.968086 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45289
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:41.140391 25362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:41.140863 25362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:41.140982 25362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:41.147063 25362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:41.147142 25362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:41.147233 25362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:27:41.155277 25362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.73.20:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:41.156831 25362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:41.158213 25362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:41.169906 25377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:41.169931 25379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:41.170168 25376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:41.171077 25362 server_base.cc:1034] running on GCE node
I20260810 06:27:41.171571 25362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:41.172367 25362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:41.173563 25362 hybrid_clock.cc:648] HybridClock initialized: now 1786343261173536 us; error 46 us; skew 500 ppm
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:41.176462 25362 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:27:41.177345 25362 webserver.cc:460] Webserver started at http://127.4.73.1:38583/ using document root <none> and password file <none>
I20260810 06:27:41.177932 25362 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:41.178058 25362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:41.178304 25362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:41.193133 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "92ab82068a29438f8fba398bcb38bc9a"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "50a05d0c21df981a96dc35323a74c177"
server_key_iv: "52a02206f8cc4ea96da4b2e9d5716560"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:41.193902 25362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "92ab82068a29438f8fba398bcb38bc9a"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "50a05d0c21df981a96dc35323a74c177"
server_key_iv: "52a02206f8cc4ea96da4b2e9d5716560"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:41.199205 25362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 06:27:41.232788 25392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:41.234670 25362 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260810 06:27:41.234805 25362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "92ab82068a29438f8fba398bcb38bc9a"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "50a05d0c21df981a96dc35323a74c177"
server_key_iv: "52a02206f8cc4ea96da4b2e9d5716560"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:41.234946 25362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:41.252230 25362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:41.273202 25362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:41.273494 25362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:41.274335 25362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:41.275539 25362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:41.275650 25362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:41.275738 25362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:41.275789 25362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:41.286576 25362 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:44989
I20260810 06:27:41.291730 25362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:27:41.297533 25577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:44989 every 8 connection(s)
I20260810 06:27:41.301558 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 25362
I20260810 06:27:41.301687 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:27:41.302018 4388 external_mini_cluster.cc:1428] Setting key 7a8a77260bf5b230bcf61f18105eeb5d
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:41.320284 25586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:41.300006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45289 (local address 127.4.73.1:59623)
0810 06:27:41.301679 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:41.301740 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:41.302850 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:41.304135 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:41.304145 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:41.304519 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:41.305199 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:41.305218 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.306401 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.306404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:41.307764 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:41.307776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.308587 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.308591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:41.308785 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:41.312760 (+ 3975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:41.312780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:41.314483 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:41.316893 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:41.316900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:41.316916 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:41.317259 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:41.317603 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:41.317606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:41.317608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:41.317807 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:41.318343 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:41.318350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:41.318706 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 06:27:41.318924 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 06:27:41.319251 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"thread_start_us":190,"threads_started":1}
I20260810 06:27:41.320586 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:41.299459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:59623 (local address 127.4.73.62:45289)
0810 06:27:41.303595 (+ 4136us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:41.303600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:41.303614 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:27:41.303653 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:41.303656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:41.303700 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:41.303788 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:41.305372 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.306248 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.307919 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.308262 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.308395 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:41.314677 (+ 6282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:41.314681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:41.314684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:41.314711 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:41.316746 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:41.317387 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:41.317391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:41.317393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:41.317491 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:41.317955 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:41.317959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:41.317961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:41.318193 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:41.318287 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:41.320263 (+ 1976us) server_negotiation.cc:299] Negotiation successful
0810 06:27:41.320426 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4058,"thread_start_us":97,"threads_started":1}
I20260810 06:27:41.322338 25582 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45289
I20260810 06:27:41.322582 25582 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:41.323182 25582 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:41.325057 25226 ts_manager.cc:194] Registered new tserver with Master: 92ab82068a29438f8fba398bcb38bc9a (127.4.73.1:44989)
I20260810 06:27:41.326584 25226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:59623
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:27:41.369206 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45289
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:41.545691 25591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:41.546152 25591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:41.546301 25591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:41.552479 25591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:41.552623 25591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:41.552765 25591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:27:41.560927 25591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.73.20:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:41.562613 25591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:41.564028 25591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:27:41.577219 25591 server_base.cc:1034] running on GCE node
W20260810 06:27:41.584867 25606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:41.585260 25603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:41.586782 25602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:41.587417 25591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:41.588282 25591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:41.595300 25591 hybrid_clock.cc:648] HybridClock initialized: now 1786343261595221 us; error 81 us; skew 500 ppm
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:41.599526 25591 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:27:41.600800 25591 webserver.cc:460] Webserver started at http://127.4.73.2:41557/ using document root <none> and password file <none>
I20260810 06:27:41.601624 25591 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:41.601828 25591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:41.602165 25591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:41.621373 25591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "5a55f77e20584cd486993bfa8a262a9b"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "4d5dfc2f902a26b3625c699605ced086"
server_key_iv: "0fc661c16775c059a3e577fce9031676"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:41.622349 25591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "5a55f77e20584cd486993bfa8a262a9b"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "4d5dfc2f902a26b3625c699605ced086"
server_key_iv: "0fc661c16775c059a3e577fce9031676"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:41.626593 25591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:27:41.643370 25620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:41.645272 25591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:27:41.645509 25591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "5a55f77e20584cd486993bfa8a262a9b"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "4d5dfc2f902a26b3625c699605ced086"
server_key_iv: "0fc661c16775c059a3e577fce9031676"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:41.645768 25591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:41.679044 25591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:41.684253 25591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:41.684559 25591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:41.691231 25591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:41.692476 25591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:41.692603 25591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:41.692710 25591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:41.692780 25591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:41.705853 25591 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:37993
I20260810 06:27:41.707149 25591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:27:41.708829 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 25591
I20260810 06:27:41.708946 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:27:41.709265 4388 external_mini_cluster.cc:1428] Setting key 6777d605ba000c99487643bc2fe4faac
I20260810 06:27:41.713887 25789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:37993 every 8 connection(s)
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:41.730353 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:41.712173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:41801 (local address 127.4.73.62:45289)
0810 06:27:41.712349 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:41.712355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:41.715380 (+ 3025us) server_negotiation.cc:407] Connection header received
0810 06:27:41.716588 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:41.716591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:41.716645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:41.716740 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:41.719722 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.720590 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.722387 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.722747 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.722834 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:41.725992 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:41.725995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:41.725998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:41.726026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:41.727809 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:41.728445 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:41.728448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:41.728450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:41.728544 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:41.728994 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:41.728997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:41.728999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:41.729224 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:41.729332 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:41.730043 (+ 711us) server_negotiation.cc:299] Negotiation successful
0810 06:27:41.730176 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:27:41.732182 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:41.711108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45289 (local address 127.4.73.2:41801)
0810 06:27:41.715211 (+ 4103us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:41.715272 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:41.716363 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:41.718425 (+ 2062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:41.718442 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:41.718854 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:41.719529 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:41.719552 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.720754 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.720757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:41.722206 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:41.722219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:41.723044 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:41.723049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:41.723243 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:41.723986 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:41.724008 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:41.725803 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:41.727950 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:41.727956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:41.727971 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:41.728322 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:41.728687 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:41.728690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:41.728692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:41.728870 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:41.729410 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:41.729416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:41.729735 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 06:27:41.730792 (+ 1057us) client_negotiation.cc:240] Negotiation successful
0810 06:27:41.731139 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3874,"thread_start_us":151,"threads_started":1}
I20260810 06:27:41.733700 25790 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45289
I20260810 06:27:41.733939 25790 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:41.734513 25790 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:41.735663 25226 ts_manager.cc:194] Registered new tserver with Master: 5a55f77e20584cd486993bfa8a262a9b (127.4.73.2:37993)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:27:41.736449 25226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:41801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:27:41.770195 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45289
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:27:41 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343261, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
W20260810 06:27:41.976333 25797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:41.976754 25797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:41.976893 25797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:41.983165 25797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:41.983332 25797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:41.983495 25797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:27:41.992023 25797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.73.20:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:41.993800 25797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:41.995416 25797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:42.011579 25809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:42.012477 25812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:42.012830 25808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:42.013411 25797 server_base.cc:1034] running on GCE node
I20260810 06:27:42.013890 25797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:42.014621 25797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:42.022058 25797 hybrid_clock.cc:648] HybridClock initialized: now 1786343262021979 us; error 87 us; skew 500 ppm
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:42.026851 25797 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:27:42.028084 25797 webserver.cc:460] Webserver started at http://127.4.73.3:32865/ using document root <none> and password file <none>
I20260810 06:27:42.028872 25797 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:42.029008 25797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:42.029299 25797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:42.051213 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "9f5746bd653b4a7abcdc352ea450cf83"
format_stamp: "Formatted at 2026-08-10 06:27:42 on dist-test-slave-45dx"
server_key: "640be9b5477fe910119c3978084f56dd"
server_key_iv: "2d39c514aa4cda166f1bc272d99a2c66"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:42.052523 25797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "9f5746bd653b4a7abcdc352ea450cf83"
format_stamp: "Formatted at 2026-08-10 06:27:42 on dist-test-slave-45dx"
server_key: "640be9b5477fe910119c3978084f56dd"
server_key_iv: "2d39c514aa4cda166f1bc272d99a2c66"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:42.058949 25797 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 06:27:42.077426 25827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:42.079072 25797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:27:42.079197 25797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "9f5746bd653b4a7abcdc352ea450cf83"
format_stamp: "Formatted at 2026-08-10 06:27:42 on dist-test-slave-45dx"
server_key: "640be9b5477fe910119c3978084f56dd"
server_key_iv: "2d39c514aa4cda166f1bc272d99a2c66"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:42.079321 25797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:42.125574 25797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:42.129879 25797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:42.130167 25797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:42.131028 25797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:42.132195 25797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:42.132350 25797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:42.132468 25797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:42.132543 25797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:42.154558 25797 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:43819
I20260810 06:27:42.155867 25797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:27:42.156332 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 25797
I20260810 06:27:42.156478 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:27:42.156798 4388 external_mini_cluster.cc:1428] Setting key 4e21c39f6d55c33a3bb6135222657cf7
I20260810 06:27:42.156827 26034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:43819 every 8 connection(s)
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:42.177295 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:42.159890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:48579 (local address 127.4.73.62:45289)
0810 06:27:42.160055 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:42.160061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:42.162072 (+ 2011us) server_negotiation.cc:407] Connection header received
0810 06:27:42.163568 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:42.163572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:42.163637 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:42.163749 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:42.165577 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.166523 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.168406 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.168827 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.168947 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:42.172487 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:42.172491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:42.172493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:42.172520 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:42.174571 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:42.175241 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:42.175245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:42.175247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:42.175363 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:42.175796 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:42.175799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:42.175801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:42.176093 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:42.176206 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:42.176947 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 06:27:42.177093 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:27:42.178273 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:42.160513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45289 (local address 127.4.73.3:48579)
0810 06:27:42.161868 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:42.161935 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:42.163235 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:42.163993 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:42.164002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:42.164546 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:42.165370 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:42.165393 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.166682 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.166686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:42.168248 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:42.168269 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.169013 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.169018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:42.169285 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:42.170135 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:42.170159 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:42.172252 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:42.174716 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:42.174723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:42.174737 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:42.175102 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:42.175468 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:42.175471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:42.175473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:42.175671 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:42.176240 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:42.176247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:42.176633 (+ 386us) client_negotiation.cc:769] Sending connection context
0810 06:27:42.176839 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 06:27:42.177182 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":365,"threads_started":1}
I20260810 06:27:42.179941 26035 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45289
I20260810 06:27:42.180193 26035 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:42.180851 26035 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:42.182431 25226 ts_manager.cc:194] Registered new tserver with Master: 9f5746bd653b4a7abcdc352ea450cf83 (127.4.73.3:43819)
I20260810 06:27:42.183255 25226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:48579
I20260810 06:27:42.184927 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:27:42.185647 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 25362
I20260810 06:27:42.217904 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 25591
I20260810 06:27:42.236872 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 25797
I20260810 06:27:42.254123 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 25172
I20260810 06:27:42.268985 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:45289
--webserver_interface=127.4.73.62
--webserver_port=39023
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:45289
--ranger_config_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:42.381994 26038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:42.382259 26038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:42.382299 26038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:42.386446 26038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:42.386510 26038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:42.386529 26038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:42.386546 26038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:42.386561 26038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:42.392019 26038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:45289
--ranger_config_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=39023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:42.393100 26038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:42.394098 26038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:42.402756 26044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:42.403093 26038 server_base.cc:1034] running on GCE node
W20260810 06:27:42.402772 26045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:42.402760 26048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:42.403754 26038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:42.404877 26038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:42.406046 26038 hybrid_clock.cc:648] HybridClock initialized: now 1786343262406025 us; error 33 us; skew 500 ppm
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:42.408931 26038 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:27:42.409749 26038 webserver.cc:460] Webserver started at http://127.4.73.62:39023/ using document root <none> and password file <none>
I20260810 06:27:42.410250 26038 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:42.410320 26038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:42.460381 26038 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:27:42.462644 26062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:42.463992 26038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:27:42.464120 26038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "d28ed21d029a4a459b3e39ce54d58c0c"
format_stamp: "Formatted at 2026-08-10 06:27:40 on dist-test-slave-45dx"
server_key: "05d62d475e5893d86b1ce2a8227a27c0"
server_key_iv: "2f49cf8b507f129a6572dabfff649ca2"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:42.464764 26038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:42.477131 26038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:42.480154 26038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:42.480432 26038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:42.487916 26038 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:45289
I20260810 06:27:42.487934 26154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:45289 every 8 connection(s)
I20260810 06:27:42.488987 26038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:27:42.493470 26155 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:27:42.495160 26155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Bootstrap starting.
I20260810 06:27:42.497574 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 26038
I20260810 06:27:42.498153 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:44989
--local_ip_for_outbound_sockets=127.4.73.1
--tserver_master_addrs=127.4.73.62:45289
--webserver_port=38583
--webserver_interface=127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:42.512915 26155 log.cc:824] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:42.519507 26155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:27:42.519940 26155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Bootstrap complete.
I20260810 06:27:42.524343 26155 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:42.524771 26155 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28ed21d029a4a459b3e39ce54d58c0c, State: Initialized, Role: FOLLOWER
I20260810 06:27:42.525309 26155 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:42.525422 26155 raft_consensus.cc:388] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:27:42.525504 26155 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:27:42.525620 26155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:27:42.527423 26155 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:42.527743 26155 leader_election.cc:302] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28ed21d029a4a459b3e39ce54d58c0c; no voters:
I20260810 06:27:42.528064 26155 leader_election.cc:288] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:27:42.528211 26159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:27:42.528515 26159 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [term 2 LEADER]: Becoming Leader. State: Replica: d28ed21d029a4a459b3e39ce54d58c0c, State: Running, Role: LEADER
I20260810 06:27:42.528904 26159 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } }
I20260810 06:27:42.529264 26155 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:42.530340 26162 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } } }
I20260810 06:27:42.530470 26162 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:42.531225 26163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28ed21d029a4a459b3e39ce54d58c0c. Latest consensus state: current_term: 2 leader_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28ed21d029a4a459b3e39ce54d58c0c" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45289 } } }
I20260810 06:27:42.531342 26163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:42.531672 26155 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 06:27:42.531797 26180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:27:42.538633 26180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:27:42.539527 26180 catalog_manager.cc:1259] Loaded cluster ID: 50803b5463bd4e669be31be582f99b6e
I20260810 06:27:42.539665 26180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:27:42.542311 26180 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:27:42.543125 26180 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Loaded TSK: 0
I20260810 06:27:42.544485 26180 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:27:42.651551 26157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:42.652056 26157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:42.652175 26157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:42.657568 26157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:42.657673 26157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:42.657795 26157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:27:42.664038 26157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:44989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=38583
--tserver_master_addrs=127.4.73.62:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:42.665266 26157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:42.666656 26157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:42.679283 26222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:42.679682 26221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:42.679807 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:42.680101 26157 server_base.cc:1034] running on GCE node
I20260810 06:27:42.680630 26157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:42.681432 26157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:42.682665 26157 hybrid_clock.cc:648] HybridClock initialized: now 1786343262682594 us; error 90 us; skew 500 ppm
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:42.686434 26157 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:27:42.687485 26157 webserver.cc:460] Webserver started at http://127.4.73.1:38583/ using document root <none> and password file <none>
I20260810 06:27:42.688135 26157 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:42.688230 26157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:42.715364 26157 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:27:42.718276 26238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:42.720024 26157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 06:27:42.720163 26157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "92ab82068a29438f8fba398bcb38bc9a"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "50a05d0c21df981a96dc35323a74c177"
server_key_iv: "52a02206f8cc4ea96da4b2e9d5716560"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:42.720976 26157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:42.742954 26157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:42.746956 26157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:42.747241 26157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:42.748054 26157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:42.749284 26157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:42.749361 26157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:42.749426 26157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:42.749511 26157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 06:27:42.762323 26157 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:44989
I20260810 06:27:42.763840 26157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:27:42.768414 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 26157
I20260810 06:27:42.768775 26429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:44989 every 8 connection(s)
I20260810 06:27:42.769083 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:37993
--local_ip_for_outbound_sockets=127.4.73.2
--tserver_master_addrs=127.4.73.62:45289
--webserver_port=41557
--webserver_interface=127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:42.800127 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:42.766558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:43049 (local address 127.4.73.62:45289)
0810 06:27:42.767236 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:42.767261 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:42.772924 (+ 5663us) server_negotiation.cc:407] Connection header received
0810 06:27:42.776043 (+ 3119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:42.776065 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:42.776573 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:42.776984 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:42.783807 (+ 6823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.785504 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.787174 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.787976 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.788123 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:42.793685 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:42.793692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:42.793710 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:42.793745 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:42.795976 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:42.796690 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:42.796697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:42.796703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:42.796852 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:42.797282 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:42.797285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:42.797287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:42.797803 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:42.797928 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:42.798719 (+ 791us) server_negotiation.cc:299] Negotiation successful
0810 06:27:42.799043 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":150,"threads_started":1}
I20260810 06:27:42.802438 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:42.766715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45289 (local address 127.4.73.1:43049)
0810 06:27:42.772695 (+ 5980us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:42.772757 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:42.775359 (+ 2602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:42.777100 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:42.777109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:42.782840 (+ 5731us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:42.783555 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:42.783574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.785716 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.785720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:42.787011 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:42.787024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:42.788450 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:42.788454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:42.788649 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:42.791527 (+ 2878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:42.791549 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:42.793436 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:42.796129 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:42.796135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:42.796149 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:42.796538 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:42.796978 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:42.796981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:42.796983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:42.797168 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:42.798000 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:42.798006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:42.798322 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 06:27:42.801161 (+ 2839us) client_negotiation.cc:240] Negotiation successful
0810 06:27:42.801490 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5716,"thread_start_us":165,"threads_started":1}
I20260810 06:27:42.805616 26430 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45289
I20260810 06:27:42.805887 26430 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:42.806393 26430 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
W20260810 06:27:42.808673 26430 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:45289 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 06:27:42.808817 26430 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:42.808990 26430 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:42.809970 26430 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:42.810149 26430 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
W20260810 06:27:42.810989 26430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:27:42.931380 26438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:42.931718 26438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:42.931808 26438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:42.936642 26438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:42.936733 26438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:42.936848 26438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:27:42.943923 26438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:37993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=41557
--tserver_master_addrs=127.4.73.62:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:42.945165 26438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:42.946483 26438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:42.956547 26445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:42.956580 26446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:42.956991 26449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:42.957051 26438 server_base.cc:1034] running on GCE node
I20260810 06:27:42.957419 26438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:42.958011 26438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:42.959179 26438 hybrid_clock.cc:648] HybridClock initialized: now 1786343262959168 us; error 34 us; skew 500 ppm
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:42.962816 26438 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:27:42.963951 26438 webserver.cc:460] Webserver started at http://127.4.73.2:41557/ using document root <none> and password file <none>
I20260810 06:27:42.964749 26438 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:42.964874 26438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:27:42 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:42.990846 26438 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:27:42.994238 26461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:42.995973 26438 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 06:27:42.996109 26438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "5a55f77e20584cd486993bfa8a262a9b"
format_stamp: "Formatted at 2026-08-10 06:27:41 on dist-test-slave-45dx"
server_key: "4d5dfc2f902a26b3625c699605ced086"
server_key_iv: "0fc661c16775c059a3e577fce9031676"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:42.996850 26438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:43.012409 26438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:43.016547 26438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:43.016844 26438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:43.017664 26438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:43.018975 26438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:43.019102 26438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:43.019223 26438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:43.019313 26438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:43.033075 26438 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:37993
I20260810 06:27:43.045681 26438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:27:43.040329 26671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:37993 every 8 connection(s)
I20260810 06:27:43.052165 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 26438
I20260810 06:27:43.052750 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.73.20:57795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:43819
--local_ip_for_outbound_sockets=127.4.73.3
--tserver_master_addrs=127.4.73.62:45289
--webserver_port=32865
--webserver_interface=127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:27:43 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343262, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:43.068231 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:43.050212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:40911 (local address 127.4.73.62:45289)
0810 06:27:43.050384 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:43.050390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:43.050854 (+ 464us) server_negotiation.cc:407] Connection header received
0810 06:27:43.052142 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:43.052147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:43.052219 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:43.052327 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:43.054163 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.055131 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.057092 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.057560 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.057660 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:43.063458 (+ 5798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:43.063463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:43.063466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:43.063512 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:43.065578 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:43.066278 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:43.066282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:43.066284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:43.066390 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:43.066823 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:43.066828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:43.066831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:43.067075 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:43.067163 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:43.067892 (+ 729us) server_negotiation.cc:299] Negotiation successful
0810 06:27:43.068035 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 06:27:43.070397 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:43.049007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45289 (local address 127.4.73.2:40911)
0810 06:27:43.050680 (+ 1673us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:43.050737 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:43.051828 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:43.052779 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:43.052789 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:43.053187 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:43.053967 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:43.053992 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.055332 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.055337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:43.056939 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:43.056955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.057897 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.057904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:43.058156 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:43.059428 (+ 1272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:43.059452 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:43.063227 (+ 3775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:43.065725 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:43.065733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:43.065750 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:43.066126 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:43.066507 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:43.066510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:43.066513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:43.066699 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:43.067235 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:43.067242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:43.067586 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 06:27:43.068625 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0810 06:27:43.069048 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"thread_start_us":170,"threads_started":1}
I20260810 06:27:43.072120 26672 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45289
I20260810 06:27:43.072417 26672 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:43.072999 26672 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
W20260810 06:27:43.076611 26672 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:45289 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 06:27:43.076732 26672 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:43.076908 26672 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:43.077822 26672 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:43.077989 26672 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
W20260810 06:27:43.078790 26672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:27:43.224960 26676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:43.225222 26676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:43.225267 26676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:43.229720 26676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:43.229806 26676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:43.229925 26676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:27:43.235826 26676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.73.20:57795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:43819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=32865
--tserver_master_addrs=127.4.73.62:45289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:43.237056 26676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:43.238041 26676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:43.249068 26689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:43.249964 26682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:43.250272 26683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:43.250628 26676 server_base.cc:1034] running on GCE node
I20260810 06:27:43.251057 26676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:43.251817 26676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:43.253011 26676 hybrid_clock.cc:648] HybridClock initialized: now 1786343263252976 us; error 49 us; skew 500 ppm
Aug 10 06:27:43 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343263, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:43.256711 26676 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:27:43.257808 26676 webserver.cc:460] Webserver started at http://127.4.73.3:32865/ using document root <none> and password file <none>
I20260810 06:27:43.258502 26676 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:43.258605 26676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 06:27:43 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343263, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:43.281792 26676 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260810 06:27:43.284727 26701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:43.286031 26676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:27:43.286178 26676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "9f5746bd653b4a7abcdc352ea450cf83"
format_stamp: "Formatted at 2026-08-10 06:27:42 on dist-test-slave-45dx"
server_key: "640be9b5477fe910119c3978084f56dd"
server_key_iv: "2d39c514aa4cda166f1bc272d99a2c66"
server_key_version: "kuduclusterkey@0"
I20260810 06:27:43.286752 26676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:43.302441 26676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:43.305994 26676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:43.306186 26676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:43.306870 26676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:43.307852 26676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:43.307924 26676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:43.308017 26676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:43.308069 26676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:43.319346 26676 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:43819
I20260810 06:27:43.319415 26891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:43819 every 8 connection(s)
I20260810 06:27:43.320565 26676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:27:43.323176 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 26676
Aug 10 06:27:43 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343263, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:43.336635 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:43.322728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:50625 (local address 127.4.73.62:45289)
0810 06:27:43.322912 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:43.322916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:43.324344 (+ 1428us) server_negotiation.cc:407] Connection header received
0810 06:27:43.325228 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:43.325232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:43.325298 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:43.325376 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:43.326730 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.327267 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.329015 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.329413 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.329500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:43.332832 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:43.332836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:43.332839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:43.332866 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:43.334407 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:43.334963 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:43.334967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:43.334969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:43.335052 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:43.335401 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:43.335404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:43.335406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:43.335588 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:43.335691 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:43.336286 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 06:27:43.336469 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 06:27:43.338212 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:43.323165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45289 (local address 127.4.73.3:50625)
0810 06:27:43.324164 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:43.324210 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:43.325018 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:43.325555 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:43.325561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:43.325850 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:43.326462 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:43.326588 (+ 126us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.327396 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.327399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:43.328863 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:43.328875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.329875 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.329880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:43.330088 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:43.331138 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:43.331159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:43.332646 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:43.334522 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:43.334527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:43.334540 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:43.334825 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:43.335144 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:43.335148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:43.335150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:43.335298 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:43.335692 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:43.335699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:43.335997 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 06:27:43.337017 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0810 06:27:43.337305 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":147,"threads_started":1}
I20260810 06:27:43.339614 26895 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45289
I20260810 06:27:43.339844 26895 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:43.340479 26895 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
W20260810 06:27:43.342206 26895 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:45289 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 06:27:43.342346 26895 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:43.342512 26895 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:43.343436 26895 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:43.343632 26895 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
W20260810 06:27:43.344419 26895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 06:27:43.349696 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:43.330929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.4.73.62:45289)
0810 06:27:43.331158 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:43.331163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:43.333192 (+ 2029us) server_negotiation.cc:407] Connection header received
0810 06:27:43.333358 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:43.333668 (+ 310us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:43.333730 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:43.333844 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:43.337698 (+ 3854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.338465 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.341123 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:43.341480 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:43.341588 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:43.344409 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:43.344413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:43.344415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:43.344443 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:43.346103 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:43.348292 (+ 2189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:43.348295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:43.348297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:43.348403 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:43.348747 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:43.348750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:43.348751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:43.348949 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:43.349099 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:43.349403 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 06:27:43.349532 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
Aug 10 06:27:43 dist-test-slave-45dx krb5kdc[24394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786343263, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:43.811228 26430 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:43.811626 26430 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:43.812834 26092 ts_manager.cc:194] Registered new tserver with Master: 92ab82068a29438f8fba398bcb38bc9a (127.4.73.1:44989)
I20260810 06:27:43.814543 26092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:43049
I20260810 06:27:44.079022 26672 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:44.079326 26672 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:44.080134 26092 ts_manager.cc:194] Registered new tserver with Master: 5a55f77e20584cd486993bfa8a262a9b (127.4.73.2:37993)
I20260810 06:27:44.081073 26092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:40911
Aug 10 06:27:44 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343263, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:44.344660 26895 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:44.344961 26895 heartbeater.cc:511] Master 127.4.73.62:45289 requested a full tablet report, sending...
I20260810 06:27:44.345762 26092 ts_manager.cc:194] Registered new tserver with Master: 9f5746bd653b4a7abcdc352ea450cf83 (127.4.73.3:43819)
I20260810 06:27:44.346535 26092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:50625
I20260810 06:27:44.353837 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:27:44.366981 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.355888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.4.73.62:45289)
0810 06:27:44.356681 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:44.356685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:44.356700 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:27:44.356742 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:44.356744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:44.356786 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:44.356880 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:44.357795 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.358599 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.359783 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.360808 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.360933 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.362637 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:44.362641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:44.362644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:44.362671 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:44.364507 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:44.365231 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:44.365234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:44.365236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:44.365327 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:44.365674 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:44.365677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:44.365679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:44.365905 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:44.366600 (+ 695us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:44.366677 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 06:27:44.366794 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":111,"threads_started":1}
Aug 10 06:27:44 dist-test-slave-45dx krb5kdc[24394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343263, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for HTTP/127.4.73.20@KRBTEST.COM
I20260810 06:27:44.379482 26092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:27:44.391439 26092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:27:44.419441 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.403653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:37993 (local address 127.0.0.1:36276)
0810 06:27:44.406218 (+ 2565us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:44.406265 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:44.406391 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:44.412862 (+ 6471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:44.412865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:44.412883 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:44.414081 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.414101 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.415375 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.415377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.416821 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.416832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.418931 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.418934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.419006 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.419023 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 06:27:44.419095 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 06:27:44.419233 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2334,"mutex_wait_us":32,"thread_start_us":168,"threads_started":1}
I20260810 06:27:44.419682 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.405608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43819 (local address 127.0.0.1:38640)
0810 06:27:44.407017 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:44.407028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:44.407126 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:44.413791 (+ 6665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:44.413794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:44.413812 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:44.413913 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.413920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.415353 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.415356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.417480 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.417492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.418802 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.418805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.418935 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.418976 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 06:27:44.419092 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 06:27:44.419552 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"mutex_wait_us":1153,"thread_start_us":136,"threads_started":1}
I20260810 06:27:44.420151 26951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.405791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.4.73.3:43819)
0810 06:27:44.413191 (+ 7400us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:44.413198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:44.413222 (+ 24us) server_negotiation.cc:407] Connection header received
0810 06:27:44.413295 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:44.413300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:44.413478 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:44.413612 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:44.414065 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.415218 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.417636 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.418443 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.418578 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.418734 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:44.419928 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0810 06:27:44.420009 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7287,"thread_start_us":97,"threads_started":1}
I20260810 06:27:44.420537 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.403761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.4.73.2:37993)
0810 06:27:44.412508 (+ 8747us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:44.412514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:44.412534 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:27:44.412593 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:44.412597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:44.412733 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:44.412891 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:44.414212 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.415268 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.416971 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.418622 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.418718 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.418876 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:44.419904 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0810 06:27:44.420003 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8654,"thread_start_us":129,"threads_started":1}
I20260810 06:27:44.423399 26768 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf0e9baf74f48f0afa9bc846d36aca8 (DEFAULT_TABLE table=test-table [id=60c2d437cf1a4aa8b5b8eb36f5d32779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:44.423341 26529 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf0e9baf74f48f0afa9bc846d36aca8 (DEFAULT_TABLE table=test-table [id=60c2d437cf1a4aa8b5b8eb36f5d32779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:44.424463 26768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf0e9baf74f48f0afa9bc846d36aca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:44.424465 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf0e9baf74f48f0afa9bc846d36aca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:44.437109 26952 tablet_bootstrap.cc:492] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Bootstrap starting.
I20260810 06:27:44.438555 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.404880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:44989 (local address 127.0.0.1:60216)
0810 06:27:44.406695 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:44.406707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:44.406806 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:44.408734 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:44.408739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:44.408764 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:44.420982 (+ 12218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.420994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.435628 (+ 14634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.435631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.436810 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.436822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.438137 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.438140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.438224 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.438247 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:27:44.438346 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 06:27:44.438419 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1749,"thread_start_us":126,"threads_started":1}
I20260810 06:27:44.439319 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.405045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.4.73.1:44989)
0810 06:27:44.407855 (+ 2810us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:44.407867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:44.407893 (+ 26us) server_negotiation.cc:407] Connection header received
0810 06:27:44.407970 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:44.407975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:44.408167 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:44.420236 (+ 12069us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:44.421121 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.435494 (+ 14373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.436951 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.437832 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.437933 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.438077 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:44.439071 (+ 994us) server_negotiation.cc:299] Negotiation successful
0810 06:27:44.439169 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2508,"thread_start_us":61,"threads_started":1}
I20260810 06:27:44.439762 26952 tablet_bootstrap.cc:654] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:44.440886 26952 log.cc:824] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:44.441650 26953 tablet_bootstrap.cc:492] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Bootstrap starting.
I20260810 06:27:44.442720 26317 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf0e9baf74f48f0afa9bc846d36aca8 (DEFAULT_TABLE table=test-table [id=60c2d437cf1a4aa8b5b8eb36f5d32779]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:44.443754 26317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf0e9baf74f48f0afa9bc846d36aca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:44.444682 26953 tablet_bootstrap.cc:654] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:44.445811 26953 log.cc:824] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:44.447847 26953 tablet_bootstrap.cc:492] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: No bootstrap required, opened a new log
I20260810 06:27:44.448020 26953 ts_tablet_manager.cc:1397] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260810 06:27:44.456348 26952 tablet_bootstrap.cc:492] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: No bootstrap required, opened a new log
I20260810 06:27:44.456678 26952 ts_tablet_manager.cc:1397] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260810 06:27:44.457677 26953 raft_consensus.cc:348] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.457934 26953 raft_consensus.cc:374] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:44.457998 26953 raft_consensus.cc:729] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a55f77e20584cd486993bfa8a262a9b, State: Initialized, Role: FOLLOWER
I20260810 06:27:44.458560 26953 consensus_queue.cc:260] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.459900 26953 ts_tablet_manager.cc:1428] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260810 06:27:44.460305 26672 heartbeater.cc:503] Master 127.4.73.62:45289 was elected leader, sending a full tablet report...
I20260810 06:27:44.466486 26952 raft_consensus.cc:348] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.466821 26952 raft_consensus.cc:374] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:44.466921 26952 raft_consensus.cc:729] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5746bd653b4a7abcdc352ea450cf83, State: Initialized, Role: FOLLOWER
I20260810 06:27:44.467445 26952 consensus_queue.cc:260] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.468688 26952 ts_tablet_manager.cc:1428] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260810 06:27:44.468716 26963 tablet_bootstrap.cc:492] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a: Bootstrap starting.
I20260810 06:27:44.469120 26895 heartbeater.cc:503] Master 127.4.73.62:45289 was elected leader, sending a full tablet report...
I20260810 06:27:44.472561 26963 tablet_bootstrap.cc:654] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:44.474584 26963 log.cc:824] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:44.477838 26963 tablet_bootstrap.cc:492] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a: No bootstrap required, opened a new log
I20260810 06:27:44.478040 26963 ts_tablet_manager.cc:1397] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260810 06:27:44.484973 26963 raft_consensus.cc:348] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.485356 26963 raft_consensus.cc:374] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:44.485471 26963 raft_consensus.cc:729] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ab82068a29438f8fba398bcb38bc9a, State: Initialized, Role: FOLLOWER
I20260810 06:27:44.486033 26963 consensus_queue.cc:260] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.487251 26963 ts_tablet_manager.cc:1428] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260810 06:27:44.487701 26430 heartbeater.cc:503] Master 127.4.73.62:45289 was elected leader, sending a full tablet report...
W20260810 06:27:44.520102 26434 tablet.cc:2369] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:27:44.548005 26674 tablet.cc:2369] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:27:44.572657 26896 tablet.cc:2369] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:27:44.625592 26967 raft_consensus.cc:482] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:27:44.625950 26967 raft_consensus.cc:504] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.627545 26967 leader_election.cc:288] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a55f77e20584cd486993bfa8a262a9b (127.4.73.2:37993), 92ab82068a29438f8fba398bcb38bc9a (127.4.73.1:44989)
I20260810 06:27:44.639530 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.628208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:58025 (local address 127.4.73.1:44989)
0810 06:27:44.628378 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:44.628383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:44.628987 (+ 604us) server_negotiation.cc:407] Connection header received
0810 06:27:44.629319 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:44.629323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:44.629378 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:44.629429 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:44.631777 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.632528 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.635566 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.636382 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.636487 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.636561 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:44.639314 (+ 2753us) server_negotiation.cc:299] Negotiation successful
0810 06:27:44.639402 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:27:44.640367 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.627949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:44989 (local address 127.4.73.3:58025)
0810 06:27:44.628868 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:44.628882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:44.629182 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:44.629927 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:44.629930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:44.629953 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:44.630413 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.630420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.632700 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.632703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.635404 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.635416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.636662 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.636666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.636833 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.636860 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 06:27:44.639956 (+ 3096us) client_negotiation.cc:240] Negotiation successful
0810 06:27:44.640038 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":176,"threads_started":1}
I20260810 06:27:44.640890 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.628400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:46649 (local address 127.4.73.2:37993)
0810 06:27:44.628545 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:44.628551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:44.628610 (+ 59us) server_negotiation.cc:407] Connection header received
0810 06:27:44.629580 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:44.629583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:44.629634 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:44.629681 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:44.630536 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.631382 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.636993 (+ 5611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.637647 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.637728 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.637788 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:44.640681 (+ 2893us) server_negotiation.cc:299] Negotiation successful
0810 06:27:44.640771 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:27:44.642374 26332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0e9baf74f48f0afa9bc846d36aca8" candidate_uuid: "9f5746bd653b4a7abcdc352ea450cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ab82068a29438f8fba398bcb38bc9a" is_pre_election: true
I20260810 06:27:44.642782 26332 raft_consensus.cc:2454] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5746bd653b4a7abcdc352ea450cf83 in term 0.
I20260810 06:27:44.643538 26705 leader_election.cc:302] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ab82068a29438f8fba398bcb38bc9a, 9f5746bd653b4a7abcdc352ea450cf83; no voters:
I20260810 06:27:44.643879 26967 raft_consensus.cc:2781] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:27:44.643960 26967 raft_consensus.cc:482] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:27:44.644004 26967 raft_consensus.cc:3037] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:44.645740 26967 raft_consensus.cc:504] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.646131 26967 leader_election.cc:288] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [CANDIDATE]: Term 1 election: Requested vote from peers 5a55f77e20584cd486993bfa8a262a9b (127.4.73.2:37993), 92ab82068a29438f8fba398bcb38bc9a (127.4.73.1:44989)
I20260810 06:27:44.647027 26332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0e9baf74f48f0afa9bc846d36aca8" candidate_uuid: "9f5746bd653b4a7abcdc352ea450cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92ab82068a29438f8fba398bcb38bc9a"
I20260810 06:27:44.647156 26332 raft_consensus.cc:3037] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:44.648736 26332 raft_consensus.cc:2454] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f5746bd653b4a7abcdc352ea450cf83 in term 1.
I20260810 06:27:44.649137 26705 leader_election.cc:302] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92ab82068a29438f8fba398bcb38bc9a, 9f5746bd653b4a7abcdc352ea450cf83; no voters:
I20260810 06:27:44.649363 26967 raft_consensus.cc:2781] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:27:44.649578 26967 raft_consensus.cc:686] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 1 LEADER]: Becoming Leader. State: Replica: 9f5746bd653b4a7abcdc352ea450cf83, State: Running, Role: LEADER
I20260810 06:27:44.649919 26967 consensus_queue.cc:237] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } }
I20260810 06:27:44.653563 26091 catalog_manager.cc:5495] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f5746bd653b4a7abcdc352ea450cf83 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "9f5746bd653b4a7abcdc352ea450cf83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f5746bd653b4a7abcdc352ea450cf83" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43819 } health_report { overall_health: HEALTHY } } }
I20260810 06:27:44.641561 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.627755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:37993 (local address 127.4.73.3:46649)
0810 06:27:44.628511 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:44.628528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:44.629076 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:44.630115 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:44.630118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:44.630132 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:44.630238 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.630255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.631557 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.631560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.634404 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:44.634416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.639000 (+ 4584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.639004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:44.639155 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.639177 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:27:44.641276 (+ 2099us) client_negotiation.cc:240] Negotiation successful
0810 06:27:44.641368 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":195,"threads_started":1}
I20260810 06:27:44.656849 26587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0e9baf74f48f0afa9bc846d36aca8" candidate_uuid: "9f5746bd653b4a7abcdc352ea450cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55f77e20584cd486993bfa8a262a9b" is_pre_election: true
I20260810 06:27:44.657141 26587 raft_consensus.cc:2454] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f5746bd653b4a7abcdc352ea450cf83 in term 0.
I20260810 06:27:44.657557 26586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf0e9baf74f48f0afa9bc846d36aca8" candidate_uuid: "9f5746bd653b4a7abcdc352ea450cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a55f77e20584cd486993bfa8a262a9b"
I20260810 06:27:44.657660 26586 raft_consensus.cc:3037] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:44.659384 26586 raft_consensus.cc:2454] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f5746bd653b4a7abcdc352ea450cf83 in term 1.
I20260810 06:27:44.707350 26951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.701613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.4.73.3:43819)
0810 06:27:44.701764 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:44.701769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:44.701789 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:27:44.702032 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:44.702035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:44.702110 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:44.702212 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:27:44.702633 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.703423 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.704764 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:44.705112 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:44.705217 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:44.705792 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:27:44.706496 (+ 704us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:27:44.706611 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:44.707034 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 06:27:44.707099 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:27:44.715240 26332 raft_consensus.cc:1261] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 92ab82068a29438f8fba398bcb38bc9a [term 1 FOLLOWER]: Refusing update from remote peer 9f5746bd653b4a7abcdc352ea450cf83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:44.716135 26974 consensus_queue.cc:1035] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92ab82068a29438f8fba398bcb38bc9a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:44.717172 26586 raft_consensus.cc:1261] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 1 FOLLOWER]: Refusing update from remote peer 9f5746bd653b4a7abcdc352ea450cf83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:44.717799 26974 consensus_queue.cc:1035] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a55f77e20584cd486993bfa8a262a9b" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:44.729176 26986 mvcc.cc:205] Tried to move back new op lower bound from 7316862012264525824 to 7316862012009291776. Current Snapshot: MvccSnapshot[applied={T|T < 7316862012264525824}]
I20260810 06:27:44.735045 26993 mvcc.cc:205] Tried to move back new op lower bound from 7316862012264525824 to 7316862012009291776. Current Snapshot: MvccSnapshot[applied={T|T < 7316862012264525824}]
I20260810 06:27:44.735621 26991 mvcc.cc:205] Tried to move back new op lower bound from 7316862012264525824 to 7316862012009291776. Current Snapshot: MvccSnapshot[applied={T|T < 7316862012264525824 or (T in {7316862012264525824})}]
I20260810 06:27:44.740216 26092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:27:44.740411 26092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:27:44.743428 26092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d28ed21d029a4a459b3e39ce54d58c0c: Sending DeleteTablet for 3 replicas of tablet 1bf0e9baf74f48f0afa9bc846d36aca8
I20260810 06:27:44.744915 26529 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf0e9baf74f48f0afa9bc846d36aca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:44 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:36276
I20260810 06:27:44.745628 26768 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf0e9baf74f48f0afa9bc846d36aca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:44 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38640
I20260810 06:27:44.746143 26317 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf0e9baf74f48f0afa9bc846d36aca8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:44 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60216
I20260810 06:27:44.747624 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 26157
I20260810 06:27:44.762055 27005 tablet_replica.cc:330] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: stopping tablet replica
I20260810 06:27:44.762459 27005 raft_consensus.cc:2229] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:27:44.762859 27005 raft_consensus.cc:2258] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:27:44.763294 27007 tablet_replica.cc:330] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: stopping tablet replica
I20260810 06:27:44.763607 27007 raft_consensus.cc:2229] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:27:44.763952 27007 raft_consensus.cc:2258] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:27:44.770108 27007 ts_tablet_manager.cc:1905] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:27:44.773183 27007 ts_tablet_manager.cc:1918] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:27:44.773300 27007 log.cc:1196] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/1bf0e9baf74f48f0afa9bc846d36aca8
I20260810 06:27:44.773638 27007 ts_tablet_manager.cc:1939] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 9f5746bd653b4a7abcdc352ea450cf83: Deleting consensus metadata
I20260810 06:27:44.774897 26063 catalog_manager.cc:4841] TS 9f5746bd653b4a7abcdc352ea450cf83 (127.4.73.3:43819): tablet 1bf0e9baf74f48f0afa9bc846d36aca8 (table test-table [id=60c2d437cf1a4aa8b5b8eb36f5d32779]) successfully deleted
I20260810 06:27:44.776088 27005 ts_tablet_manager.cc:1905] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:27:44.778756 27005 ts_tablet_manager.cc:1918] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:27:44.778865 27005 log.cc:1196] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/1bf0e9baf74f48f0afa9bc846d36aca8
I20260810 06:27:44.779220 27005 ts_tablet_manager.cc:1939] T 1bf0e9baf74f48f0afa9bc846d36aca8 P 5a55f77e20584cd486993bfa8a262a9b: Deleting consensus metadata
I20260810 06:27:44.780709 26063 catalog_manager.cc:4841] TS 5a55f77e20584cd486993bfa8a262a9b (127.4.73.2:37993): tablet 1bf0e9baf74f48f0afa9bc846d36aca8 (table test-table [id=60c2d437cf1a4aa8b5b8eb36f5d32779]) successfully deleted
W20260810 06:27:44.784694 26067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:44989 (error 108)
I20260810 06:27:44.785780 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 26438
I20260810 06:27:44.804167 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 26676
I20260810 06:27:44.807868 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:44.805179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:44989 (local address 127.0.0.1:60228)
0810 06:27:44.807706 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:44.807784 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2434}
W20260810 06:27:44.808116 26067 catalog_manager.cc:4584] TS 92ab82068a29438f8fba398bcb38bc9a (127.4.73.1:44989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1bf0e9baf74f48f0afa9bc846d36aca8: Network error: Client connection negotiation failed: client connection to 127.4.73.1:44989: connect: Connection refused (error 111)
I20260810 06:27:44.816506 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 26038
I20260810 06:27:44.845811 4388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 06:27:45.195345 4388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 06:27:45.195461 4388 mini_ranger.cc:66] Stopping Ranger...
I20260810 06:27:45.539811 4388 mini_ranger.cc:68] Stopped Ranger
2026-08-10 06:27:45.540 UTC [24454] LOG: received smart shutdown request
2026-08-10 06:27:45.544 UTC [24454] LOG: background worker "logical replication launcher" (PID 24473) exited with exit code 1
2026-08-10 06:27:45.544 UTC [24460] LOG: shutting down
2026-08-10 06:27:45.705 UTC [24454] LOG: database system is shut down
2026-08-10T06:27:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49918 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:27:45 dist-test-slave-45dx krb5kdc[27025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:27:45 dist-test-slave-45dx krb5kdc[27025](info): set up 2 sockets
Aug 10 06:27:45 dist-test-slave-45dx krb5kdc[27025](info): commencing operation
krb5kdc: starting...
W20260810 06:27:47.104745 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:27:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:27:47.249398 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34573
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34573,127.4.73.61:39691,127.4.73.60:44235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:47.357376 27042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:47.357627 27042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:47.357664 27042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:47.361534 27042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:47.361586 27042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:47.361605 27042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:47.361621 27042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:47.361636 27042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:47.366690 27042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34573,127.4.73.61:39691,127.4.73.60:44235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:47.367592 27042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:47.368532 27042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:47.377090 27053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:47.377131 27042 server_base.cc:1034] running on GCE node
W20260810 06:27:47.377096 27049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:47.377107 27050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:47.378093 27042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:47.379202 27042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:47.380396 27042 hybrid_clock.cc:648] HybridClock initialized: now 1786343267380378 us; error 30 us; skew 500 ppm
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:47.383172 27042 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:27:47.384092 27042 webserver.cc:460] Webserver started at http://127.4.73.62:44405/ using document root <none> and password file <none>
I20260810 06:27:47.384723 27042 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:47.384822 27042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:47.385073 27042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:47.387163 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "753e921d11ee483a9381c3ffea487a14"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "f113b61aa3446698da9464f996b7c19a"
server_key_iv: "24547d716bc9ccdf170c761b2db980fe"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.387823 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "753e921d11ee483a9381c3ffea487a14"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "f113b61aa3446698da9464f996b7c19a"
server_key_iv: "24547d716bc9ccdf170c761b2db980fe"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.391589 27042 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:27:47.394073 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:47.395387 27042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:27:47.395531 27042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "753e921d11ee483a9381c3ffea487a14"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "f113b61aa3446698da9464f996b7c19a"
server_key_iv: "24547d716bc9ccdf170c761b2db980fe"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.395643 27042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:47.423362 27042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:47.426563 27042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:47.426771 27042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:47.433676 27042 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:34573
I20260810 06:27:47.433754 27161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:34573 every 8 connection(s)
I20260810 06:27:47.434640 27042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:27:47.435379 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 27042
I20260810 06:27:47.435500 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:27:47.435765 4388 external_mini_cluster.cc:1428] Setting key db399c30896e4cb2f0be4ed3bc9debb0
I20260810 06:27:47.437685 27162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:47.441951 27162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.73.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.61@KRBTEST.COM" created.
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:47.457270 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.445341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.4.73.62:34573)
0810 06:27:47.445629 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.445640 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.445684 (+ 44us) server_negotiation.cc:407] Connection header received
0810 06:27:47.446138 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.446169 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.446388 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.446575 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.447463 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.448660 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.449647 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.450019 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.450192 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.452685 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.452696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.452704 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.452724 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.454291 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.454749 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.454756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.454762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.454879 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.455195 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.455198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.455200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.455533 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.455668 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.456014 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.456244 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260810 06:27:47.457268 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.444313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34573 (local address 127.0.0.1:41576)
0810 06:27:47.444892 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.444935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.445800 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.446575 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.446580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.446995 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.447301 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.447323 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.448781 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.448784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.449522 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.449531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.450171 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.450175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.450281 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.450853 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.450869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.452547 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.454403 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.454413 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.454422 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.454639 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.454974 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.454979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.454981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.455097 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.455653 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.455658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.455793 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.455955 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.456367 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":106,"thread_start_us":148,"threads_started":1}
I20260810 06:27:47.459749 27162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:27:47.460772 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.460451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58416)
0810 06:27:47.460568 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.460641 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 06:27:47.461035 27070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
I20260810 06:27:47.462049 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.461702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58426)
0810 06:27:47.461850 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.461902 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 06:27:47.462198 27162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:39691: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab.
Entry for principal kudu/127.4.73.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab.
I20260810 06:27:47.464488 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:39691
--webserver_interface=127.4.73.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab
--principal=kudu/127.4.73.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34573,127.4.73.61:39691,127.4.73.60:44235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:47.511471 27162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } attempt: 1
I20260810 06:27:47.512455 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.512111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58442)
0810 06:27:47.512269 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.512371 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 06:27:47.513315 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.513085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58450)
0810 06:27:47.513204 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.513254 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 06:27:47.513495 27162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:39691: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
W20260810 06:27:47.576941 27175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:47.577195 27175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:47.577270 27175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:47.581382 27175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:47.581435 27175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:47.581456 27175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:47.581493 27175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:47.581525 27175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:47.581663 27162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } attempt: 2
I20260810 06:27:47.582518 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.582211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58460)
0810 06:27:47.582370 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.582442 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 06:27:47.583317 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.583094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58464)
0810 06:27:47.583203 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.583250 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 06:27:47.583498 27162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:39691: Network error: Client connection negotiation failed: client connection to 127.4.73.61:39691: connect: Connection refused (error 111)
I20260810 06:27:47.587026 27175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34573,127.4.73.61:39691,127.4.73.60:44235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab
--principal=kudu/127.4.73.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:39691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.73.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:47.588084 27175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:47.589140 27175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:47.597575 27183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:47.597635 27184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:47.597658 27186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:47.597777 27175 server_base.cc:1034] running on GCE node
I20260810 06:27:47.598371 27175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:47.599512 27175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:47.600672 27175 hybrid_clock.cc:648] HybridClock initialized: now 1786343267600646 us; error 36 us; skew 500 ppm
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:47.603518 27175 init.cc:377] Logged in from keytab as kudu/127.4.73.61@KRBTEST.COM (short username kudu)
I20260810 06:27:47.604410 27175 webserver.cc:460] Webserver started at http://127.4.73.61:44269/ using document root <none> and password file <none>
I20260810 06:27:47.604892 27175 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:47.604950 27175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:47.605103 27175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:47.606967 27175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data/instance:
uuid: "3825cdd03a74480b9647c0dc99606dab"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "c714ae246f418fe37e0ba26b222b7d6f"
server_key_iv: "9116602037f32accba48b241b1f33576"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.607467 27175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/wal/instance:
uuid: "3825cdd03a74480b9647c0dc99606dab"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "c714ae246f418fe37e0ba26b222b7d6f"
server_key_iv: "9116602037f32accba48b241b1f33576"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.610992 27175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:27:47.613359 27195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:47.614596 27175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:47.614766 27175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/wal
uuid: "3825cdd03a74480b9647c0dc99606dab"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "c714ae246f418fe37e0ba26b222b7d6f"
server_key_iv: "9116602037f32accba48b241b1f33576"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.614888 27175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:47.631678 27175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:47.634809 27175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:47.635001 27175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:47.642016 27175 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.61:39691
I20260810 06:27:47.642042 27273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.61:39691 every 8 connection(s)
I20260810 06:27:47.642933 27175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
I20260810 06:27:47.645629 27274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:47.649286 27274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:27:47.650873 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 27175
I20260810 06:27:47.651036 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-1/wal/instance
I20260810 06:27:47.651384 4388 external_mini_cluster.cc:1428] Setting key ed3e840e456ba5c95421884108015745
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.60@KRBTEST.COM" created.
I20260810 06:27:47.663898 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.651170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.4.73.62:34573)
0810 06:27:47.651349 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.651354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.652257 (+ 903us) server_negotiation.cc:407] Connection header received
0810 06:27:47.653678 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.653682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.653740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.653853 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.654980 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.655505 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.656764 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.657064 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.657180 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.660049 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.660052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.660054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.660080 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.661685 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.662269 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.662275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.662277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.662365 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.662694 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.662697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.662699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.662932 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.663013 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.663583 (+ 570us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.663687 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 06:27:47.664750 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.651553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34573 (local address 127.0.0.1:41588)
0810 06:27:47.652055 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.652113 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.653421 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.654019 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.654026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.654352 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.654828 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.654840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.655646 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.655650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.656631 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.656641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.657220 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.657224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.657391 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.658007 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.658023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.659856 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.661802 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.661806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.661835 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.662079 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.662461 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.662464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.662465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.662599 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.663034 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.663039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.663308 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.663529 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.663843 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":131,"threads_started":1}
I20260810 06:27:47.666280 27274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
I20260810 06:27:47.674659 27162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } attempt: 3
I20260810 06:27:47.675333 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.666830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58466)
0810 06:27:47.666950 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.666963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.667036 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.667565 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.667568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.667736 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.667824 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.667829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.668846 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.668851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.669386 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.669394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.669978 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.669982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.670146 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.670683 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.670693 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.671870 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.673712 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.673717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.673719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.673932 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.674235 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.674240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.674241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.674359 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.674846 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.674849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.674932 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.675037 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.675161 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":36}
I20260810 06:27:47.675803 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.666955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.4.73.61:39691)
0810 06:27:47.667239 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.667248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.667265 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:27:47.667296 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.667300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.667442 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.667581 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.667932 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.668735 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.669503 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.669867 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.669982 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.672002 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.672010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.672021 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.672040 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.673604 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.674057 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.674062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.674064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.674145 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.674470 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.674473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.674475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.674732 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.674873 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.675114 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.675650 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260810 06:27:47.677774 27274 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:27:47.678572 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.678300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38394)
0810 06:27:47.678416 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.678495 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab.
Entry for principal kudu/127.4.73.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab.
W20260810 06:27:47.679284 27197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
I20260810 06:27:47.680363 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:44235
--webserver_interface=127.4.73.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab
--principal=kudu/127.4.73.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34573,127.4.73.61:39691,127.4.73.60:44235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:47.680558 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.680234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38398)
0810 06:27:47.680407 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.680478 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260810 06:27:47.680691 27274 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:44235: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
I20260810 06:27:47.685086 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.675225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58470 (local address 127.4.73.61:39691)
0810 06:27:47.675822 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.675830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.675841 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:27:47.675868 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.675870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.675954 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.676043 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.677022 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.677644 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.678640 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.679099 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.679220 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.681693 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.681696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.681698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.681719 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.683237 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.683779 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.683781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.683783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.683860 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.684165 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.684167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.684168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.684381 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.684458 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.684794 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.684883 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":164,"threads_started":1}
I20260810 06:27:47.685149 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.675201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58470)
0810 06:27:47.675332 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.675347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.675438 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.676095 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.676098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.676272 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.676898 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.676909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.677767 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.677771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.678490 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.678501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.679242 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.679249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.679414 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.679927 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.679942 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.681547 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.683343 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.683348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.683353 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.683655 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.683951 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.683957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.683962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.684070 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.684475 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.684481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.684608 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.684756 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.684932 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 06:27:47.685958 27162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:27:47.686650 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.686427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38414)
0810 06:27:47.686543 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.686590 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 06:27:47.687331 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.687081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38418)
0810 06:27:47.687220 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.687259 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260810 06:27:47.687539 27162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:44235: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
I20260810 06:27:47.718698 27162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } attempt: 1
I20260810 06:27:47.719537 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.719213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38422)
0810 06:27:47.719354 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.719420 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 06:27:47.720237 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.720022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38428)
0810 06:27:47.720131 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.720184 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 06:27:47.720391 27162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:44235: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
I20260810 06:27:47.729928 27274 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } attempt: 1
I20260810 06:27:47.730636 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.730399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38436)
0810 06:27:47.730523 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.730574 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 06:27:47.731299 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.731099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38452)
0810 06:27:47.731200 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.731247 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 06:27:47.731420 27274 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:44235: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
W20260810 06:27:47.792052 27283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:47.792351 27283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:47.792436 27283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 06:27:47.795543 27162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } attempt: 2
W20260810 06:27:47.796345 27283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:47.796411 27283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260810 06:27:47.796415 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.796054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38456)
0810 06:27:47.796197 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.796256 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 06:27:47.796445 27283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:47.796473 27283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:47.796504 27283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:47.797134 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.796933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38466)
0810 06:27:47.797041 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.797083 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 06:27:47.797254 27162 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:44235: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
I20260810 06:27:47.799567 27274 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } attempt: 2
I20260810 06:27:47.800263 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.799977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38472)
0810 06:27:47.800102 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.800183 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 06:27:47.800989 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.800788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38476)
0810 06:27:47.800895 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.800939 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 06:27:47.801124 27274 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:44235: Network error: Client connection negotiation failed: client connection to 127.4.73.60:44235: connect: Connection refused (error 111)
I20260810 06:27:47.802449 27283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34573,127.4.73.61:39691,127.4.73.60:44235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab
--principal=kudu/127.4.73.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:44235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.73.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:47.803485 27283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:47.804567 27283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:47.814121 27291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:47.814425 27283 server_base.cc:1034] running on GCE node
W20260810 06:27:47.814121 27293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:47.814141 27297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:47.815393 27283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:47.816656 27283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:47.817863 27283 hybrid_clock.cc:648] HybridClock initialized: now 1786343267817848 us; error 25 us; skew 500 ppm
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:47.820698 27283 init.cc:377] Logged in from keytab as kudu/127.4.73.60@KRBTEST.COM (short username kudu)
I20260810 06:27:47.821650 27283 webserver.cc:460] Webserver started at http://127.4.73.60:33943/ using document root <none> and password file <none>
I20260810 06:27:47.822254 27283 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:47.822379 27283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:47.822628 27283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:47.824858 27283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data/instance:
uuid: "93a694cfe11b4bb6ab2c434704be8425"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "9f5be79cb06064d5b51671d4787900c4"
server_key_iv: "426225e6bb3f28dadb39566b1d293dc4"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.825495 27283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/wal/instance:
uuid: "93a694cfe11b4bb6ab2c434704be8425"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "9f5be79cb06064d5b51671d4787900c4"
server_key_iv: "426225e6bb3f28dadb39566b1d293dc4"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.829473 27283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 06:27:47.831885 27313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:47.833088 27283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:27:47.833222 27283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/wal
uuid: "93a694cfe11b4bb6ab2c434704be8425"
format_stamp: "Formatted at 2026-08-10 06:27:47 on dist-test-slave-45dx"
server_key: "9f5be79cb06064d5b51671d4787900c4"
server_key_iv: "426225e6bb3f28dadb39566b1d293dc4"
server_key_version: "encryptionkey@0"
I20260810 06:27:47.833339 27283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:47.869490 27283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:47.873162 27283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:47.873402 27283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:47.881443 27283 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.60:44235
I20260810 06:27:47.881460 27404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.60:44235 every 8 connection(s)
I20260810 06:27:47.882454 27283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
I20260810 06:27:47.885155 27406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:47.887933 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 27283
I20260810 06:27:47.888098 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786343096561320-4388-0/minicluster-data/master-2/wal/instance
I20260810 06:27:47.888443 4388 external_mini_cluster.cc:1428] Setting key b571cdb69a4a4eff9f3c5bfe52532aee
I20260810 06:27:47.888922 27406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:27:47.892347 27274 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } attempt: 3
I20260810 06:27:47.896466 27162 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } attempt: 3
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
I20260810 06:27:47.904266 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.892770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38504)
0810 06:27:47.892907 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.892922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.893003 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.893428 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.893431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.893611 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.893715 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.893722 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.894584 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.894588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.895111 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.895120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.896360 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.896364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.896459 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.897386 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.897405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.899438 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.902056 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.902059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.902061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.902323 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.902771 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.902777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.902779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.902891 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.903686 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.903691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.903812 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.903939 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.904075 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 06:27:47.905099 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.890916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.4.73.62:34573)
0810 06:27:47.891066 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.891071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.891849 (+ 778us) server_negotiation.cc:407] Connection header received
0810 06:27:47.891996 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.891999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.892041 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.892110 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.893144 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.893863 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.895285 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.895508 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.895624 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.900333 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.900337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.900339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.900366 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.902281 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.902938 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.902941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.902943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.903033 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.903969 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.903973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.903975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.904203 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.904296 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.904810 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.904931 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 06:27:47.905893 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.890846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:34573 (local address 127.0.0.1:41598)
0810 06:27:47.891672 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.891728 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.891881 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.892204 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.892210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.892710 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.892968 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.892984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.893973 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.893977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.895126 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.895140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.895660 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.895664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.895797 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.896386 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.896414 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.900149 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.902415 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.902425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.902434 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.902815 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.903162 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.903756 (+ 594us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.903760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.903856 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.904380 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.904387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.904538 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.905305 (+ 767us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.905522 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"spinlock_wait_cycles":1408,"thread_start_us":124,"threads_started":1}
I20260810 06:27:47.905886 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.892955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.4.73.60:44235)
0810 06:27:47.893184 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.893193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.893206 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:27:47.893239 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.893241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.893277 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.893377 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.893844 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.894473 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.895963 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.896224 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.896362 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.899578 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.899583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.899593 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.899635 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.901919 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.902463 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.902469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.902471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.902653 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.903009 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.903014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.903018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.903559 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.903756 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.904615 (+ 859us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.904934 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":171,"spinlock_wait_cycles":11392,"thread_start_us":84,"threads_started":1}
I20260810 06:27:47.905886 27409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.889778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.4.73.60:44235)
0810 06:27:47.890185 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.890203 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.890239 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:27:47.891018 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.891040 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.891374 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.891596 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.892616 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.894080 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.895231 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.895603 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.895764 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.899696 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.899701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.899704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.899726 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.902463 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.902980 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.902983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.902984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.903062 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.903405 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.903408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.903410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.903598 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.904365 (+ 767us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.904533 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.904880 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":275,"spinlock_wait_cycles":1920,"thread_start_us":114,"threads_started":1}
I20260810 06:27:47.907761 27406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:27:47.909502 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.896890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38506)
0810 06:27:47.897031 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.897048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.897137 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.897685 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.897688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.897869 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.897986 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.897997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.899212 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.899215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.900336 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.900344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.902082 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.902086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.902192 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.902657 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.902676 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.904810 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.907307 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.907310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.907312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.907568 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.907925 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.907928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.907930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.908028 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.908973 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.908978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.909076 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.909168 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.909302 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 06:27:47.910076 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.897132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.4.73.60:44235)
0810 06:27:47.897425 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.897433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.897446 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:27:47.897478 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.897481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.897525 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.898107 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.898228 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.899078 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.900474 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.901932 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.903252 (+ 1320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.904951 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.904955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.904957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.904980 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.907192 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.907663 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.907666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.907667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.907799 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.908166 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.908172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.908177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.908862 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.909684 (+ 822us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.909780 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.909900 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
I20260810 06:27:47.911198 27274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab: Bootstrap starting.
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:47.915623 27162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: Bootstrap starting.
I20260810 06:27:47.917923 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.908335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.4.73.61:39691)
0810 06:27:47.908453 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.908458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.908615 (+ 157us) server_negotiation.cc:407] Connection header received
0810 06:27:47.908769 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.908772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.908818 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.908916 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.909442 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.910178 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.911190 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.911510 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.911627 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.913949 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.913953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.913956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.913983 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.915821 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.916291 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.916295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.916297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.916443 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.916844 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.916848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.916849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.917126 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.917206 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.917621 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.917753 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 06:27:47.918437 27274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:47.918514 27162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:47.918676 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.908239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:39691 (local address 127.0.0.1:58480)
0810 06:27:47.908554 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.908569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.908678 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.909029 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.909032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.909217 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.909311 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.909318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.910317 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.910320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.911054 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.911065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.911640 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.911643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.911754 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.912258 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.912271 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.913799 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.915967 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.915970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.915971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.916170 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.916596 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.916599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.916602 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.916702 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.917321 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.917325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.917420 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.918259 (+ 839us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.918434 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260810 06:27:47.919059 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.908540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.4.73.62:34573)
0810 06:27:47.908651 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.908655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.908965 (+ 310us) server_negotiation.cc:407] Connection header received
0810 06:27:47.908997 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.909000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.909039 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.909104 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.909944 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.910511 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.911330 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.911618 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.911715 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.914673 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.914678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.914681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.914705 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.916524 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.917129 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.917132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.917134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.917309 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.917619 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.917622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.917624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.917816 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.918114 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.918664 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.918924 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:27:47.919484 27162 log.cc:824] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:47.919528 27406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:27:47.920075 27274 log.cc:824] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:47.921703 27162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: No bootstrap required, opened a new log
I20260810 06:27:47.924144 27274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab: No bootstrap required, opened a new log
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
I20260810 06:27:47.926039 27162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:47.926302 27162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:47.926393 27162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753e921d11ee483a9381c3ffea487a14, State: Initialized, Role: FOLLOWER
I20260810 06:27:47.926911 27162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
Aug 10 06:27:47 dist-test-slave-45dx krb5kdc[27025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
W20260810 06:27:47.928293 27087 tablet.cc:2369] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:27:47.928982 27162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:47.929080 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.920471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.4.73.60:44235)
0810 06:27:47.920574 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.920579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.920591 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:27:47.920704 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.920710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.920756 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.920840 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.921367 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.922018 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.923000 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.923335 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.923407 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.925690 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.925694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.925696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.925718 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.927426 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.927864 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.927867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.927869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.927946 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.928213 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.928216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.928218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.928444 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.928515 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.928807 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.928936 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:27:47.929474 27413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.920248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:44235 (local address 127.0.0.1:38518)
0810 06:27:47.920496 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:47.920511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:47.920592 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:47.920946 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:47.920952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:47.921135 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:47.921248 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.921256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.922137 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.922140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.922876 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:47.922886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.923532 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.923535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:47.923611 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.924075 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:47.924089 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:47.925555 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:47.927534 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.927536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.927538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.927751 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.928043 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:47.928045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:47.928047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:47.928121 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:47.928572 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:47.928574 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:47.928650 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 06:27:47.929214 (+ 564us) client_negotiation.cc:240] Negotiation successful
0810 06:27:47.929337 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 06:27:47.931113 27274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:47.931425 27274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:47.931537 27274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3825cdd03a74480b9647c0dc99606dab, State: Initialized, Role: FOLLOWER
I20260810 06:27:47.932085 27274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:47.932670 27406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: Bootstrap starting.
I20260810 06:27:47.932709 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.920802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58494 (local address 127.4.73.61:39691)
0810 06:27:47.920924 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.920929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.920943 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:27:47.921096 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.921099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.921147 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.921235 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.921932 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.922671 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.924119 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.924480 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.924572 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.928463 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.928467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.928470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.928498 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.930564 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.931115 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.931119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.931124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.931214 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.931572 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.931575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.931578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.931819 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.931992 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.932438 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.932569 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:27:47.933761 27274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:47.935258 27406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:47.936064 27406 log.cc:824] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: Log is configured to *not* fsync() on all Append() calls
W20260810 06:27:47.936246 27437 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:27:47.936527 27437 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:27:47.934238 27042 master_runner.cc:186] Error getting master registration for 127.4.73.61:39691: OK, instance_id { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" instance_seqno: 1786343267639967 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:27:47.939426 27042 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:27:47.939699 27419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:47.939841 27419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:27:47.940048 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 06:27:47.941377 27460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 06:27:47.942497 27175 master_runner.cc:186] Error getting master registration for 127.4.73.60:44235: OK, instance_id { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" instance_seqno: 1786343267879297 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:27:47.946775 27175 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:27:47.946482 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:47.946956 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: This master's current role is: FOLLOWER
W20260810 06:27:47.947039 27460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:27:47.950333 27406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: No bootstrap required, opened a new log
I20260810 06:27:47.955250 27406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:47.955520 27406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:47.955585 27406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a694cfe11b4bb6ab2c434704be8425, State: Initialized, Role: FOLLOWER
I20260810 06:27:47.956051 27406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:47.957401 27406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:47.957530 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.943418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.4.73.61:39691)
0810 06:27:47.943523 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.943528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.946277 (+ 2749us) server_negotiation.cc:407] Connection header received
0810 06:27:47.946468 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.946474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.946526 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.946582 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.948344 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.949184 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.950421 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.950810 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.950910 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.953037 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.953041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.953045 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.953071 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.955018 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.955787 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.955791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.955793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.955895 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.956422 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.956427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.956428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.956697 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.956799 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.957177 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.957325 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 06:27:47.963251 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.942703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.4.73.62:34573)
0810 06:27:47.942836 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.942841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.944659 (+ 1818us) server_negotiation.cc:407] Connection header received
0810 06:27:47.944871 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.944875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.944923 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.944983 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.946746 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.947498 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.950929 (+ 3431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.951273 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.951371 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.957799 (+ 6428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.957804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.957807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.957838 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.959864 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.961649 (+ 1785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.961653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.961655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.961780 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.962144 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.962148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.962151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.962390 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.962517 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.962916 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.963053 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:27:47.964489 27479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:47.964613 27479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:27:47.965885 27283 master_runner.cc:263] No leader master found from master 93a694cfe11b4bb6ab2c434704be8425
I20260810 06:27:47.965953 27283 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 06:27:47.967775 27497 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:27:47.967864 27497 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:27:47.968806 27415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:47.944043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.4.73.60:44235)
0810 06:27:47.944159 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:47.944164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:47.949642 (+ 5478us) server_negotiation.cc:407] Connection header received
0810 06:27:47.949831 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:47.949834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:47.949879 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:47.949930 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:47.952554 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.953723 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.956607 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:47.956988 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:47.957083 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:47.964828 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:47.964832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:47.964835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:47.964860 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:47.966691 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.967265 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.967269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.967271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.967364 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:47.967690 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:47.967693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:47.967695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:47.967938 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:47.968100 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:47.968517 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 06:27:47.968633 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 06:27:48.124985 27321 tablet.cc:2369] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:27:48.136503 27204 tablet.cc:2369] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:27:48.194307 27439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:27:48.194518 27439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:48.195490 27439 leader_election.cc:288] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 753e921d11ee483a9381c3ffea487a14 (127.4.73.62:34573), 93a694cfe11b4bb6ab2c434704be8425 (127.4.73.60:44235)
I20260810 06:27:48.196041 27368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3825cdd03a74480b9647c0dc99606dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a694cfe11b4bb6ab2c434704be8425" is_pre_election: true
I20260810 06:27:48.196064 27118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3825cdd03a74480b9647c0dc99606dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753e921d11ee483a9381c3ffea487a14" is_pre_election: true
I20260810 06:27:48.196424 27118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3825cdd03a74480b9647c0dc99606dab in term 0.
I20260810 06:27:48.196422 27368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3825cdd03a74480b9647c0dc99606dab in term 0.
I20260810 06:27:48.196924 27197 leader_election.cc:302] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3825cdd03a74480b9647c0dc99606dab, 753e921d11ee483a9381c3ffea487a14, 93a694cfe11b4bb6ab2c434704be8425; no voters:
I20260810 06:27:48.197229 27439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:27:48.197328 27439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:27:48.197378 27439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:48.198452 27439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:48.198755 27439 leader_election.cc:288] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [CANDIDATE]: Term 1 election: Requested vote from peers 753e921d11ee483a9381c3ffea487a14 (127.4.73.62:34573), 93a694cfe11b4bb6ab2c434704be8425 (127.4.73.60:44235)
I20260810 06:27:48.199191 27118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3825cdd03a74480b9647c0dc99606dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753e921d11ee483a9381c3ffea487a14"
I20260810 06:27:48.199213 27368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3825cdd03a74480b9647c0dc99606dab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a694cfe11b4bb6ab2c434704be8425"
I20260810 06:27:48.199337 27368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:48.199344 27118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:48.200735 27368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3825cdd03a74480b9647c0dc99606dab in term 1.
I20260810 06:27:48.200735 27118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3825cdd03a74480b9647c0dc99606dab in term 1.
I20260810 06:27:48.201081 27199 leader_election.cc:302] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3825cdd03a74480b9647c0dc99606dab, 753e921d11ee483a9381c3ffea487a14, 93a694cfe11b4bb6ab2c434704be8425; no voters:
I20260810 06:27:48.201258 27439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:27:48.201468 27439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [term 1 LEADER]: Becoming Leader. State: Replica: 3825cdd03a74480b9647c0dc99606dab, State: Running, Role: LEADER
I20260810 06:27:48.201824 27439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } }
I20260810 06:27:48.203704 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3825cdd03a74480b9647c0dc99606dab. Latest consensus state: current_term: 1 leader_uuid: "3825cdd03a74480b9647c0dc99606dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:48.203831 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:48.204221 27507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:27:48.205065 27507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:27:48.209389 27368 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [term 1 FOLLOWER]: Refusing update from remote peer 3825cdd03a74480b9647c0dc99606dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:48.209425 27118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [term 1 FOLLOWER]: Refusing update from remote peer 3825cdd03a74480b9647c0dc99606dab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:48.210053 27501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:48.210276 27439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [LEADER]: Connected to new peer: Peer: permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:48.214118 27479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3825cdd03a74480b9647c0dc99606dab. Latest consensus state: current_term: 1 leader_uuid: "3825cdd03a74480b9647c0dc99606dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:48.214234 27479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:27:48.215145 27419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3825cdd03a74480b9647c0dc99606dab. Latest consensus state: current_term: 1 leader_uuid: "3825cdd03a74480b9647c0dc99606dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:48.215260 27419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:27:48.217226 27439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3825cdd03a74480b9647c0dc99606dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:48.217255 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3825cdd03a74480b9647c0dc99606dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:48.217373 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:48.217332 27439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:48.217608 27510 mvcc.cc:205] Tried to move back new op lower bound from 7316862026579595264 to 7316862026557169664. Current Snapshot: MvccSnapshot[applied={T|T < 7316862026579595264}]
I20260810 06:27:48.217855 27479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3825cdd03a74480b9647c0dc99606dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:48.217957 27479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:27:48.218430 27507 catalog_manager.cc:1347] Generated new cluster ID: 2a2ce0b9ea9a459788b1a290582c091c
I20260810 06:27:48.218489 27507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:27:48.218641 27419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3825cdd03a74480b9647c0dc99606dab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "753e921d11ee483a9381c3ffea487a14" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34573 } } peers { permanent_uuid: "3825cdd03a74480b9647c0dc99606dab" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 39691 } } peers { permanent_uuid: "93a694cfe11b4bb6ab2c434704be8425" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 44235 } } }
I20260810 06:27:48.218763 27419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:27:48.218899 27511 mvcc.cc:205] Tried to move back new op lower bound from 7316862026579595264 to 7316862026557169664. Current Snapshot: MvccSnapshot[applied={T|T < 7316862026579595264}]
I20260810 06:27:48.228819 27507 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:27:48.230032 27507 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:27:48.241786 27507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3825cdd03a74480b9647c0dc99606dab: Generated new TSK 0
I20260810 06:27:48.242589 27507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:27:48.937865 27437 catalog_manager.cc:1259] Loaded cluster ID: 2a2ce0b9ea9a459788b1a290582c091c
I20260810 06:27:48.937999 27437 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: loading cluster ID for follower catalog manager: success
I20260810 06:27:48.939632 27437 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: acquiring CA information for follower catalog manager: success
I20260810 06:27:48.940886 27437 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 753e921d11ee483a9381c3ffea487a14: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:27:48.969151 27497 catalog_manager.cc:1259] Loaded cluster ID: 2a2ce0b9ea9a459788b1a290582c091c
I20260810 06:27:48.969234 27497 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: loading cluster ID for follower catalog manager: success
I20260810 06:27:48.970875 27497 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: acquiring CA information for follower catalog manager: success
I20260810 06:27:48.971977 27497 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93a694cfe11b4bb6ab2c434704be8425: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:27:49.317488 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 27042
I20260810 06:27:49.334751 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 27175
I20260810 06:27:49.353389 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 27283
2026-08-10T06:27:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3529 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:27:49 dist-test-slave-45dx krb5kdc[27530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:27:49 dist-test-slave-45dx krb5kdc[27530](info): set up 2 sockets
Aug 10 06:27:49 dist-test-slave-45dx krb5kdc[27530](info): commencing operation
krb5kdc: starting...
W20260810 06:27:50.642649 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:27:50 dist-test-slave-45dx krb5kdc[27530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:27:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:27:50.771531 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:37051
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:37051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:50.881682 27548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:50.881955 27548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:50.881995 27548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:50.885959 27548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:50.886010 27548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:50.886029 27548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:50.886045 27548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:50.886121 27548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:50.891368 27548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:37051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:50.892410 27548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:50.893421 27548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:50.902173 27562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:50.902237 27563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:50.902192 27548 server_base.cc:1034] running on GCE node
W20260810 06:27:50.902211 27569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:50.903023 27548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:50.904168 27548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:50.905371 27548 hybrid_clock.cc:648] HybridClock initialized: now 1786343270905354 us; error 31 us; skew 500 ppm
Aug 10 06:27:50 dist-test-slave-45dx krb5kdc[27530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343270, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:50.908221 27548 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:27:50.909125 27548 webserver.cc:460] Webserver started at http://127.4.73.62:37633/ using document root <none> and password file <none>
I20260810 06:27:50.909693 27548 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:50.909829 27548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:50.910065 27548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:50.912144 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "6bb54af32960436dba25436b8ab48217"
format_stamp: "Formatted at 2026-08-10 06:27:50 on dist-test-slave-45dx"
server_key: "0c26432e9088abf3c73e90f6a0f254cd"
server_key_iv: "4c242f93f10d32d3752cb8af4e6c33f2"
server_key_version: "encryptionkey@0"
I20260810 06:27:50.912799 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "6bb54af32960436dba25436b8ab48217"
format_stamp: "Formatted at 2026-08-10 06:27:50 on dist-test-slave-45dx"
server_key: "0c26432e9088abf3c73e90f6a0f254cd"
server_key_iv: "4c242f93f10d32d3752cb8af4e6c33f2"
server_key_version: "encryptionkey@0"
I20260810 06:27:50.916821 27548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:27:50.919229 27585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:50.920468 27548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:50.920605 27548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "6bb54af32960436dba25436b8ab48217"
format_stamp: "Formatted at 2026-08-10 06:27:50 on dist-test-slave-45dx"
server_key: "0c26432e9088abf3c73e90f6a0f254cd"
server_key_iv: "4c242f93f10d32d3752cb8af4e6c33f2"
server_key_version: "encryptionkey@0"
I20260810 06:27:50.920722 27548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:50.948127 27548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:50.951058 27548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:50.951265 27548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:50.958117 27548 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:37051
I20260810 06:27:50.958142 27677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:37051 every 8 connection(s)
I20260810 06:27:50.958964 27548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:27:50.961772 27678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:50.966713 27678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217: Bootstrap starting.
I20260810 06:27:50.967563 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 27548
I20260810 06:27:50.967679 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:27:50.967931 4388 external_mini_cluster.cc:1428] Setting key 260c6904baa281d9ed14badc8ad87ee7
I20260810 06:27:50.969262 27678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:50.969993 27678 log.cc:824] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:50.972175 27678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217: No bootstrap required, opened a new log
I20260810 06:27:50.976590 27678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb54af32960436dba25436b8ab48217" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37051 } }
I20260810 06:27:50.976820 27678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:50.976897 27678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb54af32960436dba25436b8ab48217, State: Initialized, Role: FOLLOWER
Aug 10 06:27:50 dist-test-slave-45dx krb5kdc[27530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:50.977451 27678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb54af32960436dba25436b8ab48217" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37051 } }
I20260810 06:27:50.977623 27678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:27:50.977694 27678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:27:50.977792 27678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:50.978938 27678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb54af32960436dba25436b8ab48217" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37051 } }
I20260810 06:27:50.979274 27678 leader_election.cc:302] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bb54af32960436dba25436b8ab48217; no voters:
I20260810 06:27:50.979598 27678 leader_election.cc:288] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:27:50.979667 27686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:27:50.979892 27686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb54af32960436dba25436b8ab48217, State: Running, Role: LEADER
I20260810 06:27:50.980245 27686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb54af32960436dba25436b8ab48217" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37051 } }
I20260810 06:27:50.980885 27678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:50.981499 27688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bb54af32960436dba25436b8ab48217. Latest consensus state: current_term: 1 leader_uuid: "6bb54af32960436dba25436b8ab48217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb54af32960436dba25436b8ab48217" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37051 } } }
I20260810 06:27:50.981645 27688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:50.982100 27687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bb54af32960436dba25436b8ab48217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb54af32960436dba25436b8ab48217" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37051 } } }
I20260810 06:27:50.982211 27687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217 [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:50.983771 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:50.969025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.4.73.62:37051)
0810 06:27:50.969494 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:50.969508 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:50.969541 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:27:50.970171 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:50.970197 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:50.970602 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:50.970928 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:50.971818 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:50.972935 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:50.974119 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:50.974568 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:50.974690 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:50.977690 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:50.977699 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:50.977715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:50.977749 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:50.979799 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:50.980416 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:50.980422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:50.980428 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:50.980551 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:50.980875 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:50.980878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:50.980880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:50.981499 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:50.981690 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:50.982465 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 06:27:50.982787 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":310,"thread_start_us":150,"threads_started":1}
W20260810 06:27:50.986083 27706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:27:50.986195 27706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:27:50.986263 27692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:27:50.986598 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:27:50.987123 27692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:27:50.992108 27692 catalog_manager.cc:1347] Generated new cluster ID: 9af1475814ed4f57bc1de4004ef6a207
I20260810 06:27:50.992170 27692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:27:51.027645 27692 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:27:51.028939 27692 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:27:51.040632 27692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bb54af32960436dba25436b8ab48217: Generated new TSK 0
I20260810 06:27:51.041465 27692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:27:51.093669 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 27548
2026-08-10T06:27:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1741 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:27:51 dist-test-slave-45dx krb5kdc[27717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:27:51 dist-test-slave-45dx krb5kdc[27717](info): set up 2 sockets
Aug 10 06:27:51 dist-test-slave-45dx krb5kdc[27717](info): commencing operation
krb5kdc: starting...
W20260810 06:27:52.376085 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:27:52 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:27:52.503321 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43645
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:52.615486 27744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:52.615743 27744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:52.615782 27744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:52.619714 27744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:52.619774 27744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:52.619793 27744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:52.619809 27744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:52.619824 27744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:52.625077 27744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:52.626021 27744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:52.626967 27744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:52.635532 27752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:52.635555 27750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:52.635540 27755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:52.635881 27744 server_base.cc:1034] running on GCE node
I20260810 06:27:52.636489 27744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:52.637511 27744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:52.638672 27744 hybrid_clock.cc:648] HybridClock initialized: now 1786343272638654 us; error 30 us; skew 500 ppm
Aug 10 06:27:52 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343272, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:52.646903 27744 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:27:52.647800 27744 webserver.cc:460] Webserver started at http://127.4.73.62:40867/ using document root <none> and password file <none>
I20260810 06:27:52.648381 27744 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:52.648451 27744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:52.648623 27744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:52.650578 27744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "dcab673502a74f9a91adeea8736f89fa"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ef9baa707ff328ae27fe045d67b64e91"
server_key_iv: "b30dd601a348519426025e89bc76bb23"
server_key_version: "encryptionkey@0"
I20260810 06:27:52.651096 27744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "dcab673502a74f9a91adeea8736f89fa"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ef9baa707ff328ae27fe045d67b64e91"
server_key_iv: "b30dd601a348519426025e89bc76bb23"
server_key_version: "encryptionkey@0"
I20260810 06:27:52.654692 27744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:27:52.657023 27769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:52.658209 27744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:27:52.658351 27744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "dcab673502a74f9a91adeea8736f89fa"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ef9baa707ff328ae27fe045d67b64e91"
server_key_iv: "b30dd601a348519426025e89bc76bb23"
server_key_version: "encryptionkey@0"
I20260810 06:27:52.658465 27744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:52.679462 27744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:52.682209 27744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:52.682412 27744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:52.689585 27744 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43645
I20260810 06:27:52.689921 27849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43645 every 8 connection(s)
I20260810 06:27:52.690551 27744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:27:52.693339 27850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:52.698724 27850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Bootstrap starting.
I20260810 06:27:52.700163 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 27744
I20260810 06:27:52.700345 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:27:52.700614 4388 external_mini_cluster.cc:1428] Setting key c5b1805a55d902840dd42e774d9c64bb
I20260810 06:27:52.701326 27850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:52.702342 27850 log.cc:824] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:52.704510 27850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: No bootstrap required, opened a new log
I20260810 06:27:52.708730 27850 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:52.708956 27850 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:52.708990 27850 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcab673502a74f9a91adeea8736f89fa, State: Initialized, Role: FOLLOWER
I20260810 06:27:52.709429 27850 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:52.709595 27850 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:27:52.709669 27850 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:27:52.709800 27850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 0 FOLLOWER]: Advancing to term 1
Aug 10 06:27:52 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:52.710836 27850 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:52.711241 27850 leader_election.cc:302] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcab673502a74f9a91adeea8736f89fa; no voters:
I20260810 06:27:52.711622 27850 leader_election.cc:288] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:27:52.711716 27863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:27:52.712000 27863 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 LEADER]: Becoming Leader. State: Replica: dcab673502a74f9a91adeea8736f89fa, State: Running, Role: LEADER
I20260810 06:27:52.712355 27863 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:52.712994 27850 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:52.714004 27864 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcab673502a74f9a91adeea8736f89fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } } }
I20260810 06:27:52.714144 27864 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:52.714385 27865 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab673502a74f9a91adeea8736f89fa. Latest consensus state: current_term: 1 leader_uuid: "dcab673502a74f9a91adeea8736f89fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } } }
I20260810 06:27:52.714537 27865 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:52.716205 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:52.701701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.4.73.62:43645)
0810 06:27:52.702180 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:52.702206 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:52.702253 (+ 47us) server_negotiation.cc:407] Connection header received
0810 06:27:52.703012 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:52.703057 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:52.703435 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:52.703785 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:52.704514 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:52.706024 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:52.706837 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:52.707659 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:52.707835 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:52.710602 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:52.710608 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:52.710624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:52.710663 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:52.712772 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:52.713280 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:52.713286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:52.713292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:52.713442 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:52.713777 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:52.713783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:52.713787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:52.714363 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:52.714564 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:52.715011 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 06:27:52.715323 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":165,"threads_started":1}
W20260810 06:27:52.718003 27884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:27:52.718082 27884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:27:52.718176 27875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:27:52.718998 27875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:27:52.724422 27875 catalog_manager.cc:1347] Generated new cluster ID: 26ac541aeb504c1f9d5417653bacd9f2
I20260810 06:27:52.724493 27875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:27:52.740123 27875 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:27:52.741387 27875 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:27:52.754460 27875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Generated new TSK 0
I20260810 06:27:52.755190 27875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:27:52.789332 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43645
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:52.903097 27890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:52.903324 27890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:52.903401 27890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:52.907397 27890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:52.907451 27890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:52.907557 27890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:27:52.912528 27890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:52.913508 27890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:52.914620 27890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:52.924064 27899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:52.924077 27897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:52.924077 27902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:52.924314 27890 server_base.cc:1034] running on GCE node
I20260810 06:27:52.924818 27890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:52.925427 27890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:52.926616 27890 hybrid_clock.cc:648] HybridClock initialized: now 1786343272926607 us; error 27 us; skew 500 ppm
Aug 10 06:27:52 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343272, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:52.929414 27890 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:27:52.930231 27890 webserver.cc:460] Webserver started at http://127.4.73.1:38031/ using document root <none> and password file <none>
I20260810 06:27:52.930765 27890 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:52.930881 27890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:52.931104 27890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:52.933149 27890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "a09c407511b04fe9a9450d31f14a0616"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ac6b475094099754c018b20db891c7dd"
server_key_iv: "69729bab475b932471c01b3523ae9251"
server_key_version: "encryptionkey@0"
I20260810 06:27:52.933735 27890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "a09c407511b04fe9a9450d31f14a0616"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ac6b475094099754c018b20db891c7dd"
server_key_iv: "69729bab475b932471c01b3523ae9251"
server_key_version: "encryptionkey@0"
I20260810 06:27:52.937258 27890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:27:52.939531 27917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:52.940702 27890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:52.940860 27890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "a09c407511b04fe9a9450d31f14a0616"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ac6b475094099754c018b20db891c7dd"
server_key_iv: "69729bab475b932471c01b3523ae9251"
server_key_version: "encryptionkey@0"
I20260810 06:27:52.940968 27890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:52.962100 27890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:52.965381 27890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:52.965579 27890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:52.966135 27890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:52.966931 27890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:52.967002 27890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:52.967072 27890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:52.967129 27890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:52.975957 27890 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:37679
I20260810 06:27:52.976001 28117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:37679 every 8 connection(s)
I20260810 06:27:52.976809 27890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
Aug 10 06:27:52 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343272, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:52.985885 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 27890
I20260810 06:27:52.986025 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:27:52.986330 4388 external_mini_cluster.cc:1428] Setting key 86416d7abe23bd7eea32982792bbedf7
I20260810 06:27:52.990319 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:52.978571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:52803 (local address 127.4.73.62:43645)
0810 06:27:52.978696 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:52.978700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:52.979907 (+ 1207us) server_negotiation.cc:407] Connection header received
0810 06:27:52.980820 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:52.980823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:52.980868 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:52.980963 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:52.982005 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:52.982532 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:52.983555 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:52.983778 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:52.983852 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:52.986245 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:52.986248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:52.986250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:52.986273 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:52.987967 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:52.988607 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:52.988609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:52.988611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:52.988685 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:52.989073 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:52.989076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:52.989077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:52.989237 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:52.989307 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:52.989966 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 06:27:52.990106 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:27:52.991000 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:52.978924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:52803)
0810 06:27:52.979759 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:52.979799 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:52.980614 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:52.981092 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:52.981097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:52.981407 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:52.981872 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:52.981885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:52.982640 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:52.982642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:52.983461 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:52.983469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:52.983910 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:52.983912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:52.984045 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:52.984588 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:52.984603 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:52.986087 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:52.988092 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:52.988098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:52.988111 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:52.988517 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:52.988777 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:52.988782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:52.988787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:52.988977 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:52.989335 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:52.989344 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:52.989677 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 06:27:52.989875 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 06:27:52.990184 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":515,"threads_started":1}
I20260810 06:27:52.992141 28119 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43645
I20260810 06:27:52.992421 28119 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:52.992918 28119 heartbeater.cc:511] Master 127.4.73.62:43645 requested a full tablet report, sending...
I20260810 06:27:52.994298 27795 ts_manager.cc:194] Registered new tserver with Master: a09c407511b04fe9a9450d31f14a0616 (127.4.73.1:37679)
I20260810 06:27:52.995342 27795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:52803
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:27:53.013334 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43645
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:53.120748 28126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:53.120967 28126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:53.121037 28126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:53.124977 28126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:53.125025 28126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:53.125128 28126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:27:53.130033 28126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:53.130918 28126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:53.131841 28126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:53.140690 28141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:53.140735 28139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:53.140633 28138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:53.140856 28126 server_base.cc:1034] running on GCE node
I20260810 06:27:53.141312 28126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:53.141945 28126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:53.143126 28126 hybrid_clock.cc:648] HybridClock initialized: now 1786343273143114 us; error 35 us; skew 500 ppm
Aug 10 06:27:53 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343273, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:53.145900 28126 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:27:53.146713 28126 webserver.cc:460] Webserver started at http://127.4.73.2:35475/ using document root <none> and password file <none>
I20260810 06:27:53.147260 28126 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:53.147344 28126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:53.147558 28126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:53.149415 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "91dea911ddb34e3a99387017780da2cd"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "d5fc151671a4847288abeb88d37ff0d4"
server_key_iv: "35671af3504353ac5f74eb8f75134316"
server_key_version: "encryptionkey@0"
I20260810 06:27:53.149994 28126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "91dea911ddb34e3a99387017780da2cd"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "d5fc151671a4847288abeb88d37ff0d4"
server_key_iv: "35671af3504353ac5f74eb8f75134316"
server_key_version: "encryptionkey@0"
I20260810 06:27:53.153606 28126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:27:53.155939 28161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:53.157166 28126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:53.157296 28126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "91dea911ddb34e3a99387017780da2cd"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "d5fc151671a4847288abeb88d37ff0d4"
server_key_iv: "35671af3504353ac5f74eb8f75134316"
server_key_version: "encryptionkey@0"
I20260810 06:27:53.157405 28126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:53.179494 28126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:53.182801 28126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:53.183007 28126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:53.183575 28126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:53.184449 28126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:53.184513 28126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:53.184566 28126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:53.184600 28126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:53.193965 28126 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:32801
I20260810 06:27:53.193998 28349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:32801 every 8 connection(s)
I20260810 06:27:53.194841 28126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:27:53.199373 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 28126
I20260810 06:27:53.199507 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:27:53.199772 4388 external_mini_cluster.cc:1428] Setting key ffd63f3c5b8eae58a281c1a2f955dafe
Aug 10 06:27:53 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343273, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:53.209728 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.196669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:45801 (local address 127.4.73.62:43645)
0810 06:27:53.196811 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.196815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.198080 (+ 1265us) server_negotiation.cc:407] Connection header received
0810 06:27:53.198957 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.198960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.199008 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.199123 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:53.200603 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.201310 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.202534 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.202755 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.202883 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.205808 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:53.205814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:53.205820 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:53.205856 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:53.207535 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:53.208086 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:53.208091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:53.208093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:53.208180 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:53.208605 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:53.208611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:53.208613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:53.208797 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:53.208929 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.209420 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.209558 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 06:27:53.210505 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.197063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:45801)
0810 06:27:53.197924 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.197965 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:53.198737 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:53.199260 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:53.199267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:53.199760 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:53.200446 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.200464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.201450 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.201454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.202390 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.202401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.202920 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.202924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.203178 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.203869 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:53.203894 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:53.205634 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:53.207643 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:53.207648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:53.207659 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:53.207942 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:53.208283 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:53.208286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:53.208288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:53.208501 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:53.208937 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:53.208946 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:53.209175 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 06:27:53.209352 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 06:27:53.209695 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":532,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:27:53.211771 28351 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43645
I20260810 06:27:53.212016 28351 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:53.212618 28351 heartbeater.cc:511] Master 127.4.73.62:43645 requested a full tablet report, sending...
I20260810 06:27:53.213730 27795 ts_manager.cc:194] Registered new tserver with Master: 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:27:53.214403 27795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:45801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:27:53.229645 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43645
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:53.344147 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:53.344414 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:53.344460 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:53.348580 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:53.348636 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:53.348693 28358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:27:53.353634 28358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:53.354580 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:53.355566 28358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:53.365314 28367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:53.365324 28372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:53.365322 28368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:53.365360 28358 server_base.cc:1034] running on GCE node
I20260810 06:27:53.365902 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:53.366562 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:53.367749 28358 hybrid_clock.cc:648] HybridClock initialized: now 1786343273367738 us; error 36 us; skew 500 ppm
Aug 10 06:27:53 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343273, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:53.370574 28358 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:27:53.371510 28358 webserver.cc:460] Webserver started at http://127.4.73.3:41841/ using document root <none> and password file <none>
I20260810 06:27:53.372085 28358 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:53.372182 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:53.372478 28358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:27:53.374497 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "143c4969c21f42a48e6678feff8cb44e"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "9736ebd1c85ae7532a9e99b2f3a2dad0"
server_key_iv: "be64c7dc6be8f3fcd02b8cd3f0eb5ba4"
server_key_version: "encryptionkey@0"
I20260810 06:27:53.375079 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "143c4969c21f42a48e6678feff8cb44e"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "9736ebd1c85ae7532a9e99b2f3a2dad0"
server_key_iv: "be64c7dc6be8f3fcd02b8cd3f0eb5ba4"
server_key_version: "encryptionkey@0"
I20260810 06:27:53.378780 28358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:27:53.381237 28383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:53.382548 28358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:53.382691 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "143c4969c21f42a48e6678feff8cb44e"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "9736ebd1c85ae7532a9e99b2f3a2dad0"
server_key_iv: "be64c7dc6be8f3fcd02b8cd3f0eb5ba4"
server_key_version: "encryptionkey@0"
I20260810 06:27:53.382798 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:53.399906 28358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:53.403303 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:53.403520 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:53.404181 28358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:53.405084 28358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:53.405161 28358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:53.405236 28358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:53.405272 28358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:53.414533 28358 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:43905
I20260810 06:27:53.414559 28572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:43905 every 8 connection(s)
I20260810 06:27:53.415458 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:27:53.416155 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 28358
I20260810 06:27:53.416286 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:27:53.416574 4388 external_mini_cluster.cc:1428] Setting key bd1cc1fbe270cd7900b4b398d988f0fa
Aug 10 06:27:53 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343273, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:53.429143 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.417394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:34485 (local address 127.4.73.62:43645)
0810 06:27:53.417552 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.417558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.418592 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 06:27:53.419963 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.419965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.420013 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.420087 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:53.421280 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.421849 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.422976 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.423197 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.423339 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.425848 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:53.425851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:53.425853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:53.425880 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:53.427191 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:53.427663 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:53.427666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:53.427667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:53.427770 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:53.428112 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:53.428115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:53.428116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:53.428354 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:53.428459 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.428923 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.429025 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:27:53.430169 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.417891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:34485)
0810 06:27:53.418441 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.418495 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:53.419753 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:53.420217 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:53.420223 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:53.420644 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:53.421137 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.421151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.421966 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.421969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.422856 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.422865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.423371 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.423374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.423515 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.424079 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:53.424096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:53.425692 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:53.427314 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:53.427319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:53.427330 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:53.427576 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:53.427874 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:53.427876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:53.427877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:53.428004 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:53.428469 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:53.428474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:53.428702 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 06:27:53.428861 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 06:27:53.429100 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":159,"threads_started":1}
I20260810 06:27:53.431331 28575 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43645
I20260810 06:27:53.431567 28575 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:53.432062 28575 heartbeater.cc:511] Master 127.4.73.62:43645 requested a full tablet report, sending...
I20260810 06:27:53.433099 27795 ts_manager.cc:194] Registered new tserver with Master: 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
I20260810 06:27:53.433651 27795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:34485
I20260810 06:27:53.442663 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:27:53.452256 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.444796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.4.73.62:43645)
0810 06:27:53.444934 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.444940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.445149 (+ 209us) server_negotiation.cc:407] Connection header received
0810 06:27:53.445324 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.445327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.445365 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.445448 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:53.446231 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.446735 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.447584 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.447791 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.447896 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.449341 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:53.449345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:53.449348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:53.449379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:53.450563 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:53.451006 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:53.451010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:53.451011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:53.451099 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:53.451374 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:53.451378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:53.451379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:53.451596 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:53.451741 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.451987 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.452098 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:27:53.454850 27795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:27:53.457835 27795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:27:53.472638 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.466185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:37679 (local address 127.0.0.1:42868)
0810 06:27:53.467174 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.467228 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:53.467386 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:53.468167 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:53.468172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:53.468191 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:53.468404 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.468410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.469602 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.469607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.471051 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.471062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.472059 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.472063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.472171 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.472214 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 06:27:53.472358 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 06:27:53.472475 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":71,"spinlock_wait_cycles":7424,"thread_start_us":89,"threads_started":1}
I20260810 06:27:53.472654 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.466320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.4.73.1:37679)
0810 06:27:53.466691 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.466707 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.467745 (+ 1038us) server_negotiation.cc:407] Connection header received
0810 06:27:53.467837 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.467841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.467980 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.468157 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:53.468532 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.469474 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.471160 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.471925 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.472034 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.472172 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.472400 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.472512 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":91,"threads_started":1}
I20260810 06:27:53.472689 28590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.466252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:32801 (local address 127.0.0.1:38792)
0810 06:27:53.467174 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.467228 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:53.467386 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:53.468125 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:53.468130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:53.468154 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:53.468409 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.468419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.469914 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.469918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.470978 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.470988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.472258 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.472261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.472396 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.472422 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:27:53.472492 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 06:27:53.472554 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":77,"spinlock_wait_cycles":3072,"thread_start_us":104,"threads_started":1}
I20260810 06:27:53.472808 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.466527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43905 (local address 127.0.0.1:46076)
0810 06:27:53.467609 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.467621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:53.467706 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:53.469108 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:53.469110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:53.469130 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:53.469210 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.469215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.470560 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.470563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.471468 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.471476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.472380 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.472383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.472472 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.472492 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:27:53.472542 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 06:27:53.472595 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":91,"threads_started":1}
I20260810 06:27:53.473429 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.466986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.4.73.2:32801)
0810 06:27:53.467659 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.467672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.467695 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:27:53.467776 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.467784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.467959 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.468112 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:53.468566 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.469748 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.471107 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.471922 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.472034 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.472173 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.473150 (+ 977us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.473265 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":115,"threads_started":1}
I20260810 06:27:53.473457 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.467401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.4.73.3:43905)
0810 06:27:53.468664 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.468675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.468695 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:27:53.468763 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.468767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.468948 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.469101 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:53.469336 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.470444 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.471589 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.472200 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.472751 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.473132 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.473193 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.473302 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1165,"thread_start_us":166,"threads_started":1}
I20260810 06:27:53.475986 28014 tablet_service.cc:1467] Processing CreateTablet for tablet 8611de09172e4c408ca0b5699823408a (DEFAULT_TABLE table=test-table [id=b78851530b0f4d4a9bbf575afd85b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:53.475986 28460 tablet_service.cc:1467] Processing CreateTablet for tablet 8611de09172e4c408ca0b5699823408a (DEFAULT_TABLE table=test-table [id=b78851530b0f4d4a9bbf575afd85b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:53.475818 28236 tablet_service.cc:1467] Processing CreateTablet for tablet 8611de09172e4c408ca0b5699823408a (DEFAULT_TABLE table=test-table [id=b78851530b0f4d4a9bbf575afd85b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:53.477115 28014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8611de09172e4c408ca0b5699823408a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:53.477123 28460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8611de09172e4c408ca0b5699823408a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:53.477119 28236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8611de09172e4c408ca0b5699823408a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:53.483367 28599 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Bootstrap starting.
I20260810 06:27:53.484920 28605 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Bootstrap starting.
I20260810 06:27:53.485332 28601 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Bootstrap starting.
I20260810 06:27:53.486075 28599 tablet_bootstrap.cc:654] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:53.486969 28599 log.cc:824] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:53.487138 28605 tablet_bootstrap.cc:654] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:53.487532 28601 tablet_bootstrap.cc:654] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:53.488082 28605 log.cc:824] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:53.488933 28601 log.cc:824] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:53.489434 28599 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: No bootstrap required, opened a new log
I20260810 06:27:53.489647 28599 ts_tablet_manager.cc:1397] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:27:53.490104 28605 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: No bootstrap required, opened a new log
I20260810 06:27:53.490281 28605 ts_tablet_manager.cc:1397] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:27:53.490650 28601 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: No bootstrap required, opened a new log
I20260810 06:27:53.490865 28601 ts_tablet_manager.cc:1397] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:27:53.493541 28601 raft_consensus.cc:348] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.493533 28599 raft_consensus.cc:348] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.493788 28599 raft_consensus.cc:374] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:53.493775 28601 raft_consensus.cc:374] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:53.493901 28599 raft_consensus.cc:729] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91dea911ddb34e3a99387017780da2cd, State: Initialized, Role: FOLLOWER
I20260810 06:27:53.493984 28601 raft_consensus.cc:729] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09c407511b04fe9a9450d31f14a0616, State: Initialized, Role: FOLLOWER
I20260810 06:27:53.493865 28605 raft_consensus.cc:348] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.494122 28605 raft_consensus.cc:374] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:53.494228 28605 raft_consensus.cc:729] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143c4969c21f42a48e6678feff8cb44e, State: Initialized, Role: FOLLOWER
I20260810 06:27:53.494349 28599 consensus_queue.cc:260] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.494493 28601 consensus_queue.cc:260] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.494765 28605 consensus_queue.cc:260] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.495129 28351 heartbeater.cc:503] Master 127.4.73.62:43645 was elected leader, sending a full tablet report...
I20260810 06:27:53.495276 28119 heartbeater.cc:503] Master 127.4.73.62:43645 was elected leader, sending a full tablet report...
I20260810 06:27:53.495488 28601 ts_tablet_manager.cc:1428] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:27:53.495754 28605 ts_tablet_manager.cc:1428] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:27:53.495994 28599 ts_tablet_manager.cc:1428] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:27:53.496431 28575 heartbeater.cc:503] Master 127.4.73.62:43645 was elected leader, sending a full tablet report...
W20260810 06:27:53.666440 28577 tablet.cc:2369] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:27:53.695957 28354 tablet.cc:2369] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:27:53.728227 28122 tablet.cc:2369] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:27:53.831426 28610 raft_consensus.cc:482] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:27:53.831709 28610 raft_consensus.cc:504] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.833349 28610 leader_election.cc:288] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a09c407511b04fe9a9450d31f14a0616 (127.4.73.1:37679), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
I20260810 06:27:53.838125 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.834015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:44013 (local address 127.4.73.1:37679)
0810 06:27:53.834140 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.834145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.834196 (+ 51us) server_negotiation.cc:407] Connection header received
0810 06:27:53.834388 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.834394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.834451 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.834526 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:53.834902 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.835738 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.837011 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.837512 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.837594 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.837649 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.837899 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.837972 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:27:53.838135 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.833566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:37679 (local address 127.4.73.2:44013)
0810 06:27:53.834121 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.834138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:53.834251 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:53.834608 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:53.834611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:53.834629 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:53.834774 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.834785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.835864 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.835867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.836887 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.836897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.837635 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.837639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.837761 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.837795 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 06:27:53.837873 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:27:53.837941 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260810 06:27:53.838354 28620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.833789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43905 (local address 127.4.73.2:47131)
0810 06:27:53.834331 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.834346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:53.834485 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:53.834911 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:53.834914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:53.834929 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:53.835041 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.835046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.836178 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.836180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.837110 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:53.837122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.837904 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.837908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:53.838007 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.838030 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:27:53.838108 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:27:53.838187 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":19,"thread_start_us":75,"threads_started":1}
I20260810 06:27:53.838366 28598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.833898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:47131 (local address 127.4.73.3:43905)
0810 06:27:53.834037 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.834044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.834468 (+ 424us) server_negotiation.cc:407] Connection header received
0810 06:27:53.834641 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.834645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.834724 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.834843 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:53.835173 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.836050 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.837240 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.837720 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.837877 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.837936 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.838144 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.838235 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:27:53.839224 28045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8611de09172e4c408ca0b5699823408a" candidate_uuid: "91dea911ddb34e3a99387017780da2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09c407511b04fe9a9450d31f14a0616" is_pre_election: true
I20260810 06:27:53.839185 28497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8611de09172e4c408ca0b5699823408a" candidate_uuid: "91dea911ddb34e3a99387017780da2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143c4969c21f42a48e6678feff8cb44e" is_pre_election: true
I20260810 06:27:53.839514 28045 raft_consensus.cc:2454] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91dea911ddb34e3a99387017780da2cd in term 0.
I20260810 06:27:53.839514 28497 raft_consensus.cc:2454] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91dea911ddb34e3a99387017780da2cd in term 0.
I20260810 06:27:53.840008 28163 leader_election.cc:302] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 143c4969c21f42a48e6678feff8cb44e, 91dea911ddb34e3a99387017780da2cd, a09c407511b04fe9a9450d31f14a0616; no voters:
I20260810 06:27:53.840346 28610 raft_consensus.cc:2781] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:27:53.840440 28610 raft_consensus.cc:482] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:27:53.840473 28610 raft_consensus.cc:3037] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:53.841526 28610 raft_consensus.cc:504] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.841895 28610 leader_election.cc:288] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [CANDIDATE]: Term 1 election: Requested vote from peers a09c407511b04fe9a9450d31f14a0616 (127.4.73.1:37679), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
I20260810 06:27:53.842298 28045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8611de09172e4c408ca0b5699823408a" candidate_uuid: "91dea911ddb34e3a99387017780da2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a09c407511b04fe9a9450d31f14a0616"
I20260810 06:27:53.842373 28497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8611de09172e4c408ca0b5699823408a" candidate_uuid: "91dea911ddb34e3a99387017780da2cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143c4969c21f42a48e6678feff8cb44e"
I20260810 06:27:53.842439 28045 raft_consensus.cc:3037] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:53.842545 28497 raft_consensus.cc:3037] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:53.843832 28045 raft_consensus.cc:2454] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91dea911ddb34e3a99387017780da2cd in term 1.
I20260810 06:27:53.843891 28497 raft_consensus.cc:2454] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91dea911ddb34e3a99387017780da2cd in term 1.
I20260810 06:27:53.844202 28162 leader_election.cc:302] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91dea911ddb34e3a99387017780da2cd, a09c407511b04fe9a9450d31f14a0616; no voters:
I20260810 06:27:53.844458 28610 raft_consensus.cc:2781] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:27:53.844718 28610 raft_consensus.cc:686] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 1 LEADER]: Becoming Leader. State: Replica: 91dea911ddb34e3a99387017780da2cd, State: Running, Role: LEADER
I20260810 06:27:53.845098 28610 consensus_queue.cc:237] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:53.848630 27795 catalog_manager.cc:5495] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 91dea911ddb34e3a99387017780da2cd (127.4.73.2). New cstate: current_term: 1 leader_uuid: "91dea911ddb34e3a99387017780da2cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } health_report { overall_health: UNKNOWN } } }
I20260810 06:27:53.921234 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.917818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.4.73.2:32801)
0810 06:27:53.917968 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:53.917971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:53.917986 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:27:53.918087 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:53.918090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:53.918146 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:53.918236 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:27:53.918466 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.918987 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.919842 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:53.920065 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:53.920147 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:53.920426 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:27:53.920834 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:27:53.920938 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:53.920998 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 06:27:53.921048 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 06:27:53.927289 28045 raft_consensus.cc:1261] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Refusing update from remote peer 91dea911ddb34e3a99387017780da2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:53.927289 28497 raft_consensus.cc:1261] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Refusing update from remote peer 91dea911ddb34e3a99387017780da2cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:53.928145 28623 consensus_queue.cc:1035] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:53.928382 28610 consensus_queue.cc:1035] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:53.934337 28627 mvcc.cc:205] Tried to move back new op lower bound from 7316862049998680064 to 7316862049672544256. Current Snapshot: MvccSnapshot[applied={T|T < 7316862049998680064}]
I20260810 06:27:53.937180 28628 mvcc.cc:205] Tried to move back new op lower bound from 7316862049998680064 to 7316862049672544256. Current Snapshot: MvccSnapshot[applied={T|T < 7316862049998680064}]
I20260810 06:27:53.943097 27795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:27:53.943274 27795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:27:53.945819 27795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Sending DeleteTablet for 3 replicas of tablet 8611de09172e4c408ca0b5699823408a
I20260810 06:27:53.946591 28236 tablet_service.cc:1514] Processing DeleteTablet for tablet 8611de09172e4c408ca0b5699823408a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:53 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38792
I20260810 06:27:53.946583 28014 tablet_service.cc:1514] Processing DeleteTablet for tablet 8611de09172e4c408ca0b5699823408a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:53 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:42868
I20260810 06:27:53.946755 28460 tablet_service.cc:1514] Processing DeleteTablet for tablet 8611de09172e4c408ca0b5699823408a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:53 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:46076
I20260810 06:27:53.947081 28641 tablet_replica.cc:330] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: stopping tablet replica
I20260810 06:27:53.947260 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 27890
I20260810 06:27:53.957057 28643 tablet_replica.cc:330] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: stopping tablet replica
I20260810 06:27:53.957118 28641 raft_consensus.cc:2229] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:27:53.957501 28643 raft_consensus.cc:2229] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:27:53.957692 28641 raft_consensus.cc:2258] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:27:53.957870 28643 raft_consensus.cc:2258] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:27:53.959316 28643 ts_tablet_manager.cc:1905] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:27:53.959313 28641 ts_tablet_manager.cc:1905] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:27:53.962244 28643 ts_tablet_manager.cc:1918] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:27:53.962391 28643 log.cc:1196] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/8611de09172e4c408ca0b5699823408a
I20260810 06:27:53.962527 28641 ts_tablet_manager.cc:1918] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:27:53.962730 28641 log.cc:1196] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/8611de09172e4c408ca0b5699823408a
I20260810 06:27:53.962858 28643 ts_tablet_manager.cc:1939] T 8611de09172e4c408ca0b5699823408a P 143c4969c21f42a48e6678feff8cb44e: Deleting consensus metadata
I20260810 06:27:53.963122 28641 ts_tablet_manager.cc:1939] T 8611de09172e4c408ca0b5699823408a P 91dea911ddb34e3a99387017780da2cd: Deleting consensus metadata
I20260810 06:27:53.964279 27771 catalog_manager.cc:4841] TS 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905): tablet 8611de09172e4c408ca0b5699823408a (table test-table [id=b78851530b0f4d4a9bbf575afd85b68f]) successfully deleted
I20260810 06:27:53.964383 27772 catalog_manager.cc:4841] TS 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801): tablet 8611de09172e4c408ca0b5699823408a (table test-table [id=b78851530b0f4d4a9bbf575afd85b68f]) successfully deleted
W20260810 06:27:53.964933 27770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:37679 (error 108)
I20260810 06:27:53.965230 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 28126
I20260810 06:27:53.965960 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:53.965632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:37679 (local address 127.0.0.1:42878)
0810 06:27:53.965786 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:53.965862 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:37679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 06:27:53.967413 27770 catalog_manager.cc:4584] TS a09c407511b04fe9a9450d31f14a0616 (127.4.73.1:37679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8611de09172e4c408ca0b5699823408a: Network error: Client connection negotiation failed: client connection to 127.4.73.1:37679: connect: Connection refused (error 111)
I20260810 06:27:53.975854 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 28358
I20260810 06:27:53.987222 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 27744
I20260810 06:27:54.002636 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43645
--webserver_interface=127.4.73.62
--webserver_port=40867
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:54.113101 28647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:54.113389 28647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:54.113464 28647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:54.117857 28647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:54.117947 28647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:54.117982 28647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:54.118019 28647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:54.118052 28647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:54.123262 28647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:54.124269 28647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:54.125308 28647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:54.139904 28654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:54.139925 28659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:54.139912 28656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:54.140297 28647 server_base.cc:1034] running on GCE node
I20260810 06:27:54.140947 28647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:54.142031 28647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:54.143219 28647 hybrid_clock.cc:648] HybridClock initialized: now 1786343274143202 us; error 30 us; skew 500 ppm
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:54.146073 28647 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:27:54.146960 28647 webserver.cc:460] Webserver started at http://127.4.73.62:40867/ using document root <none> and password file <none>
I20260810 06:27:54.147538 28647 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:54.147646 28647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:54.151418 28647 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:27:54.153620 28674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.154762 28647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:27:54.154910 28647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "dcab673502a74f9a91adeea8736f89fa"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ef9baa707ff328ae27fe045d67b64e91"
server_key_iv: "b30dd601a348519426025e89bc76bb23"
server_key_version: "encryptionkey@0"
I20260810 06:27:54.155519 28647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:54.175894 28647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:54.178699 28647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:54.178936 28647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:54.186758 28647 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43645
I20260810 06:27:54.187078 28770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43645 every 8 connection(s)
I20260810 06:27:54.188151 28647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:27:54.189363 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 28647
I20260810 06:27:54.189870 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:37679
--local_ip_for_outbound_sockets=127.4.73.1
--tserver_master_addrs=127.4.73.62:43645
--webserver_port=38031
--webserver_interface=127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:54.193519 28772 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:27:54.195186 28772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Bootstrap starting.
I20260810 06:27:54.212387 28772 log.cc:824] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:54.220700 28772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:27:54.221094 28772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Bootstrap complete.
I20260810 06:27:54.225126 28772 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:54.225494 28772 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcab673502a74f9a91adeea8736f89fa, State: Initialized, Role: FOLLOWER
I20260810 06:27:54.225956 28772 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:54.226064 28772 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:27:54.226152 28772 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:27:54.226269 28772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:27:54.227988 28772 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:54.228343 28772 leader_election.cc:302] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcab673502a74f9a91adeea8736f89fa; no voters:
I20260810 06:27:54.228668 28772 leader_election.cc:288] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:27:54.228742 28777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:27:54.228915 28777 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 LEADER]: Becoming Leader. State: Replica: dcab673502a74f9a91adeea8736f89fa, State: Running, Role: LEADER
I20260810 06:27:54.229176 28777 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:54.229833 28772 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:54.230515 28778 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dcab673502a74f9a91adeea8736f89fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } } }
I20260810 06:27:54.230916 28778 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:54.230821 28779 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab673502a74f9a91adeea8736f89fa. Latest consensus state: current_term: 2 leader_uuid: "dcab673502a74f9a91adeea8736f89fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } } }
I20260810 06:27:54.231374 28779 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:54.231580 28790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:27:54.235878 28790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:27:54.236605 28790 catalog_manager.cc:1259] Loaded cluster ID: 26ac541aeb504c1f9d5417653bacd9f2
I20260810 06:27:54.236665 28790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:27:54.238009 28790 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 06:27:54.242798 28790 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 06:27:54.247781 28790 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:27:54.248483 28790 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Loaded TSK: 0
I20260810 06:27:54.249625 28790 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:27:54.319002 28775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:54.319309 28775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:54.319409 28775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:54.323555 28775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:54.323628 28775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:54.323711 28775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:27:54.328943 28775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:37679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=38031
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:54.330060 28775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:54.331238 28775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:54.341167 28815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:54.341204 28812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:54.341360 28819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:54.341403 28775 server_base.cc:1034] running on GCE node
I20260810 06:27:54.341885 28775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:54.342604 28775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:54.343802 28775 hybrid_clock.cc:648] HybridClock initialized: now 1786343274343771 us; error 40 us; skew 500 ppm
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:54.347235 28775 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:27:54.348413 28775 webserver.cc:460] Webserver started at http://127.4.73.1:38031/ using document root <none> and password file <none>
I20260810 06:27:54.349084 28775 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:54.349180 28775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:54.354110 28775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:27:54.356669 28836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.358222 28775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:27:54.358333 28775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "a09c407511b04fe9a9450d31f14a0616"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ac6b475094099754c018b20db891c7dd"
server_key_iv: "69729bab475b932471c01b3523ae9251"
server_key_version: "encryptionkey@0"
I20260810 06:27:54.359012 28775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:54.382987 28775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:54.386324 28775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:54.386511 28775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:54.387142 28775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:54.388433 28843 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:27:54.392633 28775 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:27:54.392705 28775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 06:27:54.392771 28775 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:27:54.394887 28775 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:27:54.394958 28775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 06:27:54.395251 28843 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Bootstrap starting.
I20260810 06:27:54.405436 28775 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:37679
I20260810 06:27:54.405750 29039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:37679 every 8 connection(s)
I20260810 06:27:54.406497 28775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:27:54.409353 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 28775
I20260810 06:27:54.410017 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:32801
--local_ip_for_outbound_sockets=127.4.73.2
--tserver_master_addrs=127.4.73.62:43645
--webserver_port=35475
--webserver_interface=127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:54.416348 28843 log.cc:824] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:54.425366 28843 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:27:54.425810 28843 tablet_bootstrap.cc:492] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Bootstrap complete.
I20260810 06:27:54.427125 28843 ts_tablet_manager.cc:1397] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.000s
I20260810 06:27:54.427124 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.408656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:36903 (local address 127.4.73.62:43645)
0810 06:27:54.408983 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.409020 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.410723 (+ 1703us) server_negotiation.cc:407] Connection header received
0810 06:27:54.412184 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.412201 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.412626 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.412948 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:54.414361 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.415315 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.416757 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.417231 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.417442 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.421094 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:54.421101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:54.421115 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:54.421147 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:54.423373 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.424054 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.424060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.424065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.424206 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.424610 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.424613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.424615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.425041 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:54.425162 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.425907 (+ 745us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.426227 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":119,"threads_started":1}
I20260810 06:27:54.428481 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.408672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:36903)
0810 06:27:54.410554 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.410608 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.411654 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.413006 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.413014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.413663 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:54.414160 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.414174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.415498 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.415501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.416626 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.416637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.417954 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.417958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.418148 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.418888 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:54.418910 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:54.420869 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:54.423505 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.423511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.423539 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.423909 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.424341 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.424344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.424346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.424499 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.425231 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:54.425237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:54.425535 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.427367 (+ 1832us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.427647 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1672,"thread_start_us":136,"threads_started":1}
I20260810 06:27:54.429826 28843 raft_consensus.cc:348] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:54.430403 28843 raft_consensus.cc:729] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09c407511b04fe9a9450d31f14a0616, State: Initialized, Role: FOLLOWER
I20260810 06:27:54.430903 28843 consensus_queue.cc:260] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } } peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } }
I20260810 06:27:54.431581 29040 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43645
I20260810 06:27:54.431782 29040 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:54.431991 28843 ts_tablet_manager.cc:1428] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 06:27:54.432404 29040 heartbeater.cc:511] Master 127.4.73.62:43645 requested a full tablet report, sending...
I20260810 06:27:54.434520 28701 ts_manager.cc:194] Registered new tserver with Master: a09c407511b04fe9a9450d31f14a0616 (127.4.73.1:37679)
I20260810 06:27:54.438177 28701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:36903
I20260810 06:27:54.440626 29040 heartbeater.cc:503] Master 127.4.73.62:43645 was elected leader, sending a full tablet report...
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:27:54.447971 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.437649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:37679 (local address 127.0.0.1:42882)
0810 06:27:54.438183 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.438224 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.438357 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.438979 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.438984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.439302 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:54.439400 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.439406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.440387 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.440394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.441126 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.441136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.441981 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.441998 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.442244 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.442767 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:54.442783 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:54.444446 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:54.446229 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.446239 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.446244 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.446472 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.446801 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.446804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.446806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.446894 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.447348 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:54.447354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:54.447474 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.447608 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.447801 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":99,"threads_started":1}
I20260810 06:27:54.447985 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.437753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.4.73.1:37679)
0810 06:27:54.438417 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.438429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.438522 (+ 93us) server_negotiation.cc:407] Connection header received
0810 06:27:54.438590 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.438595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.438782 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.438903 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:54.439516 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.440242 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.441289 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.441859 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.441973 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.444586 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:54.444602 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:54.444611 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:54.444633 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:54.446112 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.446601 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.446605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.446606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.446681 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.446986 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.446991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.446992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.447212 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:54.447369 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.447701 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.447851 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":89,"threads_started":1}
I20260810 06:27:54.449117 28932 tablet_service.cc:1514] Processing DeleteTablet for tablet 8611de09172e4c408ca0b5699823408a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:53 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:42882
I20260810 06:27:54.449947 29060 tablet_replica.cc:330] stopping tablet replica
I20260810 06:27:54.450256 29060 raft_consensus.cc:2229] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:27:54.450502 29060 raft_consensus.cc:2258] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:27:54.451606 29060 ts_tablet_manager.cc:1905] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:27:54.455981 29060 ts_tablet_manager.cc:1918] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:27:54.456082 29060 log.cc:1196] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/8611de09172e4c408ca0b5699823408a
I20260810 06:27:54.456418 29060 ts_tablet_manager.cc:1939] T 8611de09172e4c408ca0b5699823408a P a09c407511b04fe9a9450d31f14a0616: Deleting consensus metadata
I20260810 06:27:54.458206 28675 catalog_manager.cc:4841] TS a09c407511b04fe9a9450d31f14a0616 (127.4.73.1:37679): tablet 8611de09172e4c408ca0b5699823408a (table test-table [id=b78851530b0f4d4a9bbf575afd85b68f]) successfully deleted
W20260810 06:27:54.531610 29050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:54.531855 29050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:54.531896 29050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:54.535787 29050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:54.535841 29050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:54.535894 29050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:27:54.540894 29050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:32801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=35475
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:54.541832 29050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:54.542748 29050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:54.552124 29070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:54.552220 29050 server_base.cc:1034] running on GCE node
W20260810 06:27:54.552107 29065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:54.552122 29066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:54.552768 29050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:54.553303 29050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:54.554468 29050 hybrid_clock.cc:648] HybridClock initialized: now 1786343274554440 us; error 40 us; skew 500 ppm
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:54.557245 29050 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:27:54.558197 29050 webserver.cc:460] Webserver started at http://127.4.73.2:35475/ using document root <none> and password file <none>
I20260810 06:27:54.558810 29050 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:54.558888 29050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:54.562695 29050 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 06:27:54.564880 29084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.566156 29050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:54.566246 29050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "91dea911ddb34e3a99387017780da2cd"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "d5fc151671a4847288abeb88d37ff0d4"
server_key_iv: "35671af3504353ac5f74eb8f75134316"
server_key_version: "encryptionkey@0"
I20260810 06:27:54.566733 29050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:54.577114 29050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:54.579741 29050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:54.579895 29050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:54.580435 29050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:54.581236 29050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:54.581282 29050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.581319 29050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:54.581339 29050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.589937 29050 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:32801
I20260810 06:27:54.590003 29294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:32801 every 8 connection(s)
I20260810 06:27:54.590973 29050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:54.601162 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 29050
I20260810 06:27:54.601672 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:43905
--local_ip_for_outbound_sockets=127.4.73.3
--tserver_master_addrs=127.4.73.62:43645
--webserver_port=41841
--webserver_interface=127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:54.604794 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.592757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:58189 (local address 127.4.73.62:43645)
0810 06:27:54.592901 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.592904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.593704 (+ 800us) server_negotiation.cc:407] Connection header received
0810 06:27:54.595027 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.595031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.595089 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.595168 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:54.596283 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.596847 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.597950 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.598263 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.598391 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.600711 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:54.600715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:54.600718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:54.600744 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:54.602504 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.603026 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.603032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.603037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.603135 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.603644 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.603647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.603648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.603821 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:54.603902 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.604566 (+ 664us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.604677 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:27:54.605835 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.593128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:58189)
0810 06:27:54.593558 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.593598 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.594766 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.595326 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.595332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.595680 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:54.596134 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.596146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.596977 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.596981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.597815 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.597832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.598423 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.598427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.598598 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.599086 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:54.599101 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:54.600554 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:54.602632 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.602636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.602646 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.602922 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.603246 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.603249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.603250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.603532 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.603935 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:54.603943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:54.604279 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.604522 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.604862 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260810 06:27:54.606995 29297 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43645
I20260810 06:27:54.607237 29297 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:54.607699 29297 heartbeater.cc:511] Master 127.4.73.62:43645 requested a full tablet report, sending...
I20260810 06:27:54.608804 28701 ts_manager.cc:194] Registered new tserver with Master: 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801)
I20260810 06:27:54.609666 28701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:58189
W20260810 06:27:54.713788 29301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:54.714037 29301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:54.714092 29301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:54.718029 29301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:54.718084 29301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:54.718139 29301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:27:54.723245 29301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:43905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=41841
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:54.724186 29301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:54.725219 29301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:54.734894 29312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:54.734907 29310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:54.734899 29316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:54.735373 29301 server_base.cc:1034] running on GCE node
I20260810 06:27:54.735773 29301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:54.736433 29301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:54.737622 29301 hybrid_clock.cc:648] HybridClock initialized: now 1786343274737603 us; error 36 us; skew 500 ppm
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:54.740406 29301 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:27:54.741189 29301 webserver.cc:460] Webserver started at http://127.4.73.3:41841/ using document root <none> and password file <none>
I20260810 06:27:54.741715 29301 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:54.741799 29301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:54.745615 29301 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 06:27:54.747694 29333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.748961 29301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:27:54.749075 29301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "143c4969c21f42a48e6678feff8cb44e"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "9736ebd1c85ae7532a9e99b2f3a2dad0"
server_key_iv: "be64c7dc6be8f3fcd02b8cd3f0eb5ba4"
server_key_version: "encryptionkey@0"
I20260810 06:27:54.749634 29301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:54.772471 29301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:54.775197 29301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:54.775401 29301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:54.775976 29301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:54.776787 29301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:54.776834 29301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.776898 29301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:54.776932 29301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:54.785950 29301 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:43905
I20260810 06:27:54.785972 29520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:43905 every 8 connection(s)
I20260810 06:27:54.786897 29301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:27:54.789906 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 29301
Aug 10 06:27:54 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343274, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:27:54.798702 29526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.790604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.4.73.62:43645)
0810 06:27:54.790847 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.790852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.791038 (+ 186us) server_negotiation.cc:407] Connection header received
0810 06:27:54.791223 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.791225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.791269 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.791378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:54.792119 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.792719 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.793614 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.793847 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.793952 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.795578 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:54.795582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:54.795584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:54.795615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:54.796992 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.797461 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.797465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.797466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.797554 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.797840 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.797844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.797845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.798033 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:54.798245 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.798383 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.798503 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
I20260810 06:27:54.801733 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.788713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:46463 (local address 127.4.73.62:43645)
0810 06:27:54.788858 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.788863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.789716 (+ 853us) server_negotiation.cc:407] Connection header received
0810 06:27:54.791306 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.791309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.791363 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.791467 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:54.792806 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.793299 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.794621 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.795021 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.795150 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.797745 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:54.797749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:54.797753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:54.797781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:54.799424 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.800059 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.800062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.800064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.800148 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:54.800682 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:54.800685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:54.800686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:54.800855 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:54.800966 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.801472 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.801585 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:27:54.802525 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.789131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:46463)
0810 06:27:54.789569 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.789610 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.791086 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.791647 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.791653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.792037 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:54.792645 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.792660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.793433 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.793441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.794479 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.794493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.795180 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.795183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.795345 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.795917 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:54.795942 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:54.797577 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:54.799541 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.799546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.799557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.799938 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.800245 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:54.800249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:54.800251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:54.800451 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:54.800967 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:54.800975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:54.801221 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.801411 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.801721 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260810 06:27:54.803632 29521 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43645
I20260810 06:27:54.803915 29521 heartbeater.cc:464] Registering TS with master...
I20260810 06:27:54.804483 29521 heartbeater.cc:511] Master 127.4.73.62:43645 requested a full tablet report, sending...
I20260810 06:27:54.805580 28701 ts_manager.cc:194] Registered new tserver with Master: 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
I20260810 06:27:54.806143 28701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:46463
I20260810 06:27:54.812146 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:27:54.816461 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.812842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.4.73.62:43645)
0810 06:27:54.812986 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.812993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.813129 (+ 136us) server_negotiation.cc:407] Connection header received
0810 06:27:54.813286 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.813290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.813378 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.813465 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:27:54.813691 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.814232 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.815058 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.815291 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.815373 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.815613 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:27:54.815982 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:27:54.816076 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.816130 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.816172 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:27:54.817080 28701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:27:54.821099 28701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:27:54.828609 28932 tablet_service.cc:1467] Processing CreateTablet for tablet e3729f8a982342928952a348704ee3df (DEFAULT_TABLE table=test-table [id=9799cf069a594c8dbf3bf476c3157ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:54.829312 28932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3729f8a982342928952a348704ee3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:54.832039 28843 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Bootstrap starting.
I20260810 06:27:54.832823 28843 tablet_bootstrap.cc:654] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:54.833671 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.827290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:32801 (local address 127.0.0.1:38820)
0810 06:27:54.827386 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.827402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.827473 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.828728 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.828731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.828750 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:54.828866 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.828873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.830630 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.830634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.832089 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.832100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.833196 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.833200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.833333 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.833382 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.833457 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.833546 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":123,"spinlock_wait_cycles":1408}
I20260810 06:27:54.833809 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.827541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.4.73.2:32801)
0810 06:27:54.828234 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.828246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.828268 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:27:54.828361 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.828367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.828564 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.828725 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:54.828989 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.830378 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.832229 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.833056 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.833289 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.833464 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.833541 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.833640 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":165,"threads_started":1}
I20260810 06:27:54.834170 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.827596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43905 (local address 127.0.0.1:46092)
0810 06:27:54.827875 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.827885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.827978 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.828743 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.828746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.828764 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:54.829084 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.829091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.830614 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.830637 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.832089 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.832100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.833193 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.833196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.833868 (+ 672us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.833905 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.833975 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.834040 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":369,"thread_start_us":123,"threads_started":1}
I20260810 06:27:54.834223 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.828053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.4.73.3:43905)
0810 06:27:54.828392 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.828403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.828421 (+ 18us) server_negotiation.cc:407] Connection header received
0810 06:27:54.828476 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.828480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.828605 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.828871 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:54.829173 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.830427 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.832229 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.833056 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.833174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.833314 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.834007 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.834093 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":133,"threads_started":1}
I20260810 06:27:54.835912 28843 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: No bootstrap required, opened a new log
I20260810 06:27:54.836007 28843 ts_tablet_manager.cc:1397] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 06:27:54.836509 28843 raft_consensus.cc:348] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.836634 28843 raft_consensus.cc:374] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:54.836673 28843 raft_consensus.cc:729] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09c407511b04fe9a9450d31f14a0616, State: Initialized, Role: FOLLOWER
I20260810 06:27:54.836805 28843 consensus_queue.cc:260] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.837066 29408 tablet_service.cc:1467] Processing CreateTablet for tablet e3729f8a982342928952a348704ee3df (DEFAULT_TABLE table=test-table [id=9799cf069a594c8dbf3bf476c3157ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:54.838306 29408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3729f8a982342928952a348704ee3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:54.838186 29180 tablet_service.cc:1467] Processing CreateTablet for tablet e3729f8a982342928952a348704ee3df (DEFAULT_TABLE table=test-table [id=9799cf069a594c8dbf3bf476c3157ede]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:27:54.839133 28843 ts_tablet_manager.cc:1428] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 06:27:54.839406 29180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3729f8a982342928952a348704ee3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:27:54.846099 29542 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Bootstrap starting.
I20260810 06:27:54.846395 29543 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Bootstrap starting.
I20260810 06:27:54.848523 29542 tablet_bootstrap.cc:654] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:54.848788 29543 tablet_bootstrap.cc:654] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Neither blocks nor log segments found. Creating new log.
I20260810 06:27:54.849412 29542 log.cc:824] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:54.849649 29543 log.cc:824] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:54.851177 29542 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: No bootstrap required, opened a new log
I20260810 06:27:54.851307 29543 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: No bootstrap required, opened a new log
I20260810 06:27:54.851413 29542 ts_tablet_manager.cc:1397] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:27:54.851519 29543 ts_tablet_manager.cc:1397] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 06:27:54.854668 29543 raft_consensus.cc:348] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.854882 29543 raft_consensus.cc:374] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:54.854941 29543 raft_consensus.cc:729] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91dea911ddb34e3a99387017780da2cd, State: Initialized, Role: FOLLOWER
I20260810 06:27:54.854764 29542 raft_consensus.cc:348] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.855021 29542 raft_consensus.cc:374] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:27:54.855078 29542 raft_consensus.cc:729] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 143c4969c21f42a48e6678feff8cb44e, State: Initialized, Role: FOLLOWER
I20260810 06:27:54.855368 29543 consensus_queue.cc:260] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.855481 29542 consensus_queue.cc:260] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.856007 29297 heartbeater.cc:503] Master 127.4.73.62:43645 was elected leader, sending a full tablet report...
I20260810 06:27:54.856320 29521 heartbeater.cc:503] Master 127.4.73.62:43645 was elected leader, sending a full tablet report...
I20260810 06:27:54.856366 29543 ts_tablet_manager.cc:1428] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 06:27:54.856446 29542 ts_tablet_manager.cc:1428] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:27:54.899192 29055 raft_consensus.cc:482] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:27:54.899395 29055 raft_consensus.cc:504] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.900558 29055 leader_election.cc:288] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
I20260810 06:27:54.905637 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.900847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:59891 (local address 127.4.73.2:32801)
0810 06:27:54.900990 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.900997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.901016 (+ 19us) server_negotiation.cc:407] Connection header received
0810 06:27:54.901165 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.901169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.901228 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.901328 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:54.901666 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.902505 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.904259 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.904945 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.905056 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.905120 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.905434 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.905504 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:27:54.905637 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.901010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:35551 (local address 127.4.73.3:43905)
0810 06:27:54.901137 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.901141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.901309 (+ 168us) server_negotiation.cc:407] Connection header received
0810 06:27:54.901446 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.901450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.901500 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.901559 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:27:54.901852 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.902562 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.904259 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.904945 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.905056 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.905120 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.905434 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.905505 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:27:54.905670 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.900708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:32801 (local address 127.4.73.1:59891)
0810 06:27:54.900877 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.900895 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.900995 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.901368 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.901373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.901393 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:54.901519 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.901524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.902656 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.902660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.904114 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.904126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.905075 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.905081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.905256 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.905318 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.905402 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.905516 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":196}
I20260810 06:27:54.905682 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.900828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43905 (local address 127.4.73.1:35551)
0810 06:27:54.901198 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.901209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:54.901336 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:54.901598 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:54.901604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:54.901626 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:27:54.901728 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.901732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.902663 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.902665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.904114 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:54.904126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.905085 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.905088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:54.905261 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.905316 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 06:27:54.905402 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 06:27:54.905512 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":131,"thread_start_us":137,"threads_started":1}
I20260810 06:27:54.906360 29213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3729f8a982342928952a348704ee3df" candidate_uuid: "a09c407511b04fe9a9450d31f14a0616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91dea911ddb34e3a99387017780da2cd" is_pre_election: true
I20260810 06:27:54.906498 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3729f8a982342928952a348704ee3df" candidate_uuid: "a09c407511b04fe9a9450d31f14a0616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143c4969c21f42a48e6678feff8cb44e" is_pre_election: true
I20260810 06:27:54.906657 29213 raft_consensus.cc:2454] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09c407511b04fe9a9450d31f14a0616 in term 0.
I20260810 06:27:54.906832 29440 raft_consensus.cc:2454] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09c407511b04fe9a9450d31f14a0616 in term 0.
I20260810 06:27:54.907164 28839 leader_election.cc:302] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91dea911ddb34e3a99387017780da2cd, a09c407511b04fe9a9450d31f14a0616; no voters:
I20260810 06:27:54.907454 29055 raft_consensus.cc:2781] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:27:54.907539 29055 raft_consensus.cc:482] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:27:54.907569 29055 raft_consensus.cc:3037] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 0 FOLLOWER]: Advancing to term 1
W20260810 06:27:54.907809 29044 tablet.cc:2369] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:27:54.908886 29055 raft_consensus.cc:504] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.909216 29055 leader_election.cc:288] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 1 election: Requested vote from peers 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
I20260810 06:27:54.909533 29213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3729f8a982342928952a348704ee3df" candidate_uuid: "a09c407511b04fe9a9450d31f14a0616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91dea911ddb34e3a99387017780da2cd"
I20260810 06:27:54.909574 29440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3729f8a982342928952a348704ee3df" candidate_uuid: "a09c407511b04fe9a9450d31f14a0616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "143c4969c21f42a48e6678feff8cb44e"
I20260810 06:27:54.909665 29213 raft_consensus.cc:3037] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:54.909667 29440 raft_consensus.cc:3037] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:27:54.910971 29440 raft_consensus.cc:2454] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09c407511b04fe9a9450d31f14a0616 in term 1.
I20260810 06:27:54.910971 29213 raft_consensus.cc:2454] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09c407511b04fe9a9450d31f14a0616 in term 1.
I20260810 06:27:54.911250 28839 leader_election.cc:302] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91dea911ddb34e3a99387017780da2cd, a09c407511b04fe9a9450d31f14a0616; no voters:
I20260810 06:27:54.911430 29055 raft_consensus.cc:2781] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:27:54.911691 29055 raft_consensus.cc:686] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 LEADER]: Becoming Leader. State: Replica: a09c407511b04fe9a9450d31f14a0616, State: Running, Role: LEADER
I20260810 06:27:54.911958 29055 consensus_queue.cc:237] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:54.915226 28695 catalog_manager.cc:5495] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 reported cstate change: term changed from 0 to 1, leader changed from <none> to a09c407511b04fe9a9450d31f14a0616 (127.4.73.1). New cstate: current_term: 1 leader_uuid: "a09c407511b04fe9a9450d31f14a0616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } health_report { overall_health: HEALTHY } } }
I20260810 06:27:54.941075 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.937466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.4.73.1:37679)
0810 06:27:54.937599 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:54.937605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:54.937625 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:27:54.937837 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:54.937840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:54.937899 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:54.937994 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:27:54.938259 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.938819 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.939650 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:54.939908 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:54.939994 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:54.940227 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:27:54.940626 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:27:54.940710 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:54.940818 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 06:27:54.940905 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:27:54.944784 29440 raft_consensus.cc:1261] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Refusing update from remote peer a09c407511b04fe9a9450d31f14a0616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:54.944811 29213 raft_consensus.cc:1261] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 1 FOLLOWER]: Refusing update from remote peer a09c407511b04fe9a9450d31f14a0616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:27:54.945541 29551 consensus_queue.cc:1035] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:54.945729 29055 consensus_queue.cc:1035] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:27:54.958913 29563 mvcc.cc:205] Tried to move back new op lower bound from 7316862054167547904 to 7316862054042148864. Current Snapshot: MvccSnapshot[applied={T|T < 7316862054167547904}]
I20260810 06:27:54.961815 28701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:27:54.961988 28701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:27:54.964461 28701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Sending DeleteTablet for 3 replicas of tablet e3729f8a982342928952a348704ee3df
I20260810 06:27:54.965574 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 28775
I20260810 06:27:54.965543 29180 tablet_service.cc:1514] Processing DeleteTablet for tablet e3729f8a982342928952a348704ee3df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:54 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38820
I20260810 06:27:54.968932 29408 tablet_service.cc:1514] Processing DeleteTablet for tablet e3729f8a982342928952a348704ee3df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:27:54 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:46092
I20260810 06:27:54.970640 29571 tablet_replica.cc:330] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: stopping tablet replica
I20260810 06:27:54.970891 29571 raft_consensus.cc:2229] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:27:54.971127 29571 raft_consensus.cc:2258] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:27:54.973187 29571 ts_tablet_manager.cc:1905] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:27:54.977339 29571 ts_tablet_manager.cc:1918] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:27:54.977515 29571 log.cc:1196] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/e3729f8a982342928952a348704ee3df
I20260810 06:27:54.977892 29571 ts_tablet_manager.cc:1939] T e3729f8a982342928952a348704ee3df P 143c4969c21f42a48e6678feff8cb44e: Deleting consensus metadata
I20260810 06:27:54.978926 28676 catalog_manager.cc:4841] TS 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905): tablet e3729f8a982342928952a348704ee3df (table test-table [id=9799cf069a594c8dbf3bf476c3157ede]) successfully deleted
I20260810 06:27:54.980686 29570 tablet_replica.cc:330] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: stopping tablet replica
I20260810 06:27:54.981007 29570 raft_consensus.cc:2229] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:27:54.981295 29570 raft_consensus.cc:2258] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:27:54.982578 29570 ts_tablet_manager.cc:1905] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:27:54.984724 29570 ts_tablet_manager.cc:1918] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:27:54.984851 29570 log.cc:1196] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/e3729f8a982342928952a348704ee3df
I20260810 06:27:54.985236 29570 ts_tablet_manager.cc:1939] T e3729f8a982342928952a348704ee3df P 91dea911ddb34e3a99387017780da2cd: Deleting consensus metadata
W20260810 06:27:54.985347 28675 connection.cc:535] client connection to 127.4.73.1:37679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 06:27:54.985504 28675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:27:54.986488 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:54.986183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:37679 (local address 127.0.0.1:42900)
0810 06:27:54.986308 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:54.986398 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:37679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 06:27:54.986449 28677 catalog_manager.cc:4841] TS 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801): tablet e3729f8a982342928952a348704ee3df (table test-table [id=9799cf069a594c8dbf3bf476c3157ede]) successfully deleted
I20260810 06:27:54.986615 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 29050
W20260810 06:27:54.986754 28675 catalog_manager.cc:4584] TS a09c407511b04fe9a9450d31f14a0616 (127.4.73.1:37679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3729f8a982342928952a348704ee3df: Network error: Client connection negotiation failed: client connection to 127.4.73.1:37679: connect: Connection refused (error 111)
I20260810 06:27:54.998394 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 29301
I20260810 06:27:55.009377 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 28647
I20260810 06:27:55.023687 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43645
--webserver_interface=127.4.73.62
--webserver_port=40867
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:27:55.133296 29573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:55.133571 29573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:55.133643 29573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:55.137375 29573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:27:55.137449 29573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:55.137483 29573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:55.137535 29573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:27:55.137570 29573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:27:55.142688 29573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=40867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:55.143784 29573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:55.144848 29573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:55.153415 29582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:55.153426 29585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:55.153469 29573 server_base.cc:1034] running on GCE node
W20260810 06:27:55.153424 29590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:55.154232 29573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:55.155265 29573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:55.156378 29573 hybrid_clock.cc:648] HybridClock initialized: now 1786343275156341 us; error 40 us; skew 500 ppm
Aug 10 06:27:55 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343275, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:55.158994 29573 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:27:55.159849 29573 webserver.cc:460] Webserver started at http://127.4.73.62:40867/ using document root <none> and password file <none>
I20260810 06:27:55.160405 29573 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:55.160473 29573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:55.164098 29573 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:55.166160 29607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.167333 29573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:27:55.167457 29573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "dcab673502a74f9a91adeea8736f89fa"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ef9baa707ff328ae27fe045d67b64e91"
server_key_iv: "b30dd601a348519426025e89bc76bb23"
server_key_version: "encryptionkey@0"
I20260810 06:27:55.167969 29573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:55.192426 29573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:55.197407 29573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:55.197666 29573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:55.207352 29573 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43645
I20260810 06:27:55.207800 29690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43645 every 8 connection(s)
I20260810 06:27:55.209245 29573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:27:55.210579 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 29573
I20260810 06:27:55.211133 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:37679
--local_ip_for_outbound_sockets=127.4.73.1
--tserver_master_addrs=127.4.73.62:43645
--webserver_port=38031
--webserver_interface=127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:55.214624 29695 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:27:55.217314 29695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Bootstrap starting.
I20260810 06:27:55.242738 29695 log.cc:824] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:55.254169 29695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:27:55.254561 29695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa: Bootstrap complete.
I20260810 06:27:55.258401 29695 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:55.258780 29695 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcab673502a74f9a91adeea8736f89fa, State: Initialized, Role: FOLLOWER
I20260810 06:27:55.259272 29695 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:55.259419 29695 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:27:55.259505 29695 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:27:55.259615 29695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:27:55.261598 29695 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:55.261935 29695 leader_election.cc:302] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcab673502a74f9a91adeea8736f89fa; no voters:
I20260810 06:27:55.262245 29695 leader_election.cc:288] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 06:27:55.262370 29701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 3 FOLLOWER]: Leader election won for term 3
I20260810 06:27:55.262627 29701 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [term 3 LEADER]: Becoming Leader. State: Replica: dcab673502a74f9a91adeea8736f89fa, State: Running, Role: LEADER
I20260810 06:27:55.262908 29701 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } }
I20260810 06:27:55.263376 29695 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:27:55.263801 29701 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcab673502a74f9a91adeea8736f89fa. Latest consensus state: current_term: 3 leader_uuid: "dcab673502a74f9a91adeea8736f89fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } } }
I20260810 06:27:55.263906 29701 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:55.264251 29709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:27:55.264451 29702 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dcab673502a74f9a91adeea8736f89fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcab673502a74f9a91adeea8736f89fa" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43645 } } }
I20260810 06:27:55.264635 29702 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcab673502a74f9a91adeea8736f89fa [sys.catalog]: This master's current role is: LEADER
I20260810 06:27:55.269491 29709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:27:55.270114 29709 catalog_manager.cc:1259] Loaded cluster ID: 26ac541aeb504c1f9d5417653bacd9f2
I20260810 06:27:55.270175 29709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 06:27:55.271247 29709 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f52fd07e9ad google::LogMessage::Fail() at ??:0
@ 0x7f52fd08188a google::LogMessage::SendToLog() at ??:0
@ 0x7f52fd07e4de google::LogMessage::Flush() at ??:0
@ 0x7f52fd07fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f53079c1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f53079c2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f53079bf579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5307a03640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f52ff2d03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f52ff2d0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f52ff2d23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f52ff2c1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f53009056db start_thread at ??:0
@ 0x7f52fbb7c61f clone at ??:0
W20260810 06:27:55.353427 29698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:55.353641 29698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:55.353681 29698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:55.357532 29698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:55.357611 29698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:55.357730 29698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:27:55.362984 29698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:37679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=38031
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:55.363989 29698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:55.365007 29698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:55.374692 29769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:55.374732 29765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:55.374729 29766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:55.374956 29698 server_base.cc:1034] running on GCE node
I20260810 06:27:55.375532 29698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:55.376349 29698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:55.377565 29698 hybrid_clock.cc:648] HybridClock initialized: now 1786343275377531 us; error 42 us; skew 500 ppm
Aug 10 06:27:55 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343275, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:55.381155 29698 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:27:55.382284 29698 webserver.cc:460] Webserver started at http://127.4.73.1:38031/ using document root <none> and password file <none>
I20260810 06:27:55.382987 29698 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:55.383077 29698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:55.388098 29698 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:27:55.391616 29789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.393172 29698 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 06:27:55.393285 29698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "a09c407511b04fe9a9450d31f14a0616"
format_stamp: "Formatted at 2026-08-10 06:27:52 on dist-test-slave-45dx"
server_key: "ac6b475094099754c018b20db891c7dd"
server_key_iv: "69729bab475b932471c01b3523ae9251"
server_key_version: "encryptionkey@0"
I20260810 06:27:55.393996 29698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:55.417443 29698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:55.421226 29698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:55.421422 29698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:55.422129 29698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:55.423447 29801 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:27:55.427505 29698 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:27:55.427577 29698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 06:27:55.427639 29698 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:27:55.429834 29698 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:27:55.429898 29698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 06:27:55.430233 29801 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Bootstrap starting.
I20260810 06:27:55.441627 29698 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:37679
I20260810 06:27:55.441674 29980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:37679 every 8 connection(s)
I20260810 06:27:55.442802 29698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:27:55.449857 29801 log.cc:824] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Log is configured to *not* fsync() on all Append() calls
I20260810 06:27:55.451083 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 29698
I20260810 06:27:55.451859 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:32801
--local_ip_for_outbound_sockets=127.4.73.2
--tserver_master_addrs=127.4.73.62:43645
--webserver_port=35475
--webserver_interface=127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:55.458629 29801 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:27:55.459081 29801 tablet_bootstrap.cc:492] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Bootstrap complete.
I20260810 06:27:55.460731 29801 ts_tablet_manager.cc:1397] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260810 06:27:55.464046 29801 raft_consensus.cc:348] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:55.464602 29801 raft_consensus.cc:729] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09c407511b04fe9a9450d31f14a0616, State: Initialized, Role: FOLLOWER
I20260810 06:27:55.465124 29801 consensus_queue.cc:260] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:55.466243 29801 ts_tablet_manager.cc:1428] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260810 06:27:55.506419 29984 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:27:55.445091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:37657)
0810 06:27:55.445670 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.445726 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:55.446663 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:55.505444 (+ 58781us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":175,"threads_started":1}
W20260810 06:27:55.507108 29981 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:43645 (0 consecutive failures): Network error: Failed to ping master at 127.4.73.62:43645: Client connection negotiation failed: client connection to 127.4.73.62:43645: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:27:55.507939 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.507627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:52387)
0810 06:27:55.507772 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.507859 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 06:27:55.510948 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.510638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:51799)
0810 06:27:55.510764 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.510837 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 06:27:55.511222 29981 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:27:55.566905 29988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:55.567167 29988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:55.567236 29988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:55.571233 29988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:55.571312 29988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:55.571399 29988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:27:55.576187 29988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:32801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=35475
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:55.577140 29988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:55.578086 29988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:55.588063 30004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:55.588070 30008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:55.588105 30005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:55.588105 29988 server_base.cc:1034] running on GCE node
I20260810 06:27:55.588792 29988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:55.589464 29988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:55.590657 29988 hybrid_clock.cc:648] HybridClock initialized: now 1786343275590608 us; error 61 us; skew 500 ppm
Aug 10 06:27:55 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343275, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:55.593449 29988 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:27:55.594295 29988 webserver.cc:460] Webserver started at http://127.4.73.2:35475/ using document root <none> and password file <none>
I20260810 06:27:55.594846 29988 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:55.594939 29988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:55.598632 29988 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:55.600807 30019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.602072 29988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:27:55.602164 29988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "91dea911ddb34e3a99387017780da2cd"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "d5fc151671a4847288abeb88d37ff0d4"
server_key_iv: "35671af3504353ac5f74eb8f75134316"
server_key_version: "encryptionkey@0"
I20260810 06:27:55.602640 29988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:55.614440 29988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:55.617185 29988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:55.617329 29988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:55.617808 29988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:55.618537 29988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:55.618585 29988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.618623 29988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:55.618638 29988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.627532 29988 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:32801
I20260810 06:27:55.627568 30199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:32801 every 8 connection(s)
I20260810 06:27:55.628460 29988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:27:55.629258 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 29988
I20260810 06:27:55.629675 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:43905
--local_ip_for_outbound_sockets=127.4.73.3
--tserver_master_addrs=127.4.73.62:43645
--webserver_port=41841
--webserver_interface=127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:27:55.632406 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.630606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:54493)
0810 06:27:55.631085 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.631691 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":148,"threads_started":1}
W20260810 06:27:55.632947 30202 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:43645 (0 consecutive failures): Network error: Failed to ping master at 127.4.73.62:43645: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
I20260810 06:27:55.633520 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.633300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:37269)
0810 06:27:55.633410 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.633457 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 06:27:55.634234 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.634042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:49977)
0810 06:27:55.634147 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.634188 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 06:27:55.634428 30202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:27:55.742811 30210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:27:55.743052 30210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:27:55.743119 30210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:27:55.747102 30210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:27:55.747149 30210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:27:55.747241 30210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:27:55.752167 30210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:43905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=41841
--tserver_master_addrs=127.4.73.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:27:55.753075 30210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:27:55.753954 30210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:27:55.763499 30219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:55.763528 30225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:27:55.763528 30221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:27:55.763728 30210 server_base.cc:1034] running on GCE node
I20260810 06:27:55.764148 30210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:27:55.764765 30210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:27:55.765962 30210 hybrid_clock.cc:648] HybridClock initialized: now 1786343275765940 us; error 32 us; skew 500 ppm
Aug 10 06:27:55 dist-test-slave-45dx krb5kdc[27717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343275, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:27:55.768679 30210 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:27:55.769462 30210 webserver.cc:460] Webserver started at http://127.4.73.3:41841/ using document root <none> and password file <none>
I20260810 06:27:55.769955 30210 fs_manager.cc:359] Metadata directory not provided
I20260810 06:27:55.770013 30210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:27:55.773669 30210 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:27:55.775611 30238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.776912 30210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:27:55.777017 30210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "143c4969c21f42a48e6678feff8cb44e"
format_stamp: "Formatted at 2026-08-10 06:27:53 on dist-test-slave-45dx"
server_key: "9736ebd1c85ae7532a9e99b2f3a2dad0"
server_key_iv: "be64c7dc6be8f3fcd02b8cd3f0eb5ba4"
server_key_version: "encryptionkey@0"
I20260810 06:27:55.777529 30210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:27:55.793143 30210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:27:55.795950 30210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:27:55.796100 30210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:27:55.796650 30210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:27:55.797503 30210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:27:55.797551 30210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.797621 30210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:27:55.797659 30210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:27:55.807346 30210 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:43905
I20260810 06:27:55.807372 30425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:43905 every 8 connection(s)
I20260810 06:27:55.808385 30210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:27:55.812120 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.810490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:56173)
0810 06:27:55.810953 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.811454 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
W20260810 06:27:55.812683 30428 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:43645 (0 consecutive failures): Network error: Failed to ping master at 127.4.73.62:43645: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
I20260810 06:27:55.813309 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.813069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:49263)
0810 06:27:55.813197 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.813246 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 06:27:55.813972 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:55.813774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:60637)
0810 06:27:55.813880 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:55.813924 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 06:27:55.814177 30428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 06:27:55.817212 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 30210
I20260810 06:27:56.512149 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:56.511759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:42149)
0810 06:27:56.512000 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:56.512052 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260810 06:27:56.635391 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:56.634996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:40913)
0810 06:27:56.635232 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:56.635294 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260810 06:27:56.815094 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:56.814700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:58727)
0810 06:27:56.814931 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:56.814985 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260810 06:27:56.909279 30439 raft_consensus.cc:482] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:27:56.909564 30439 raft_consensus.cc:504] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:56.910676 30439 leader_election.cc:288] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
Aug 10 06:27:56 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343275, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.2@KRBTEST.COM
Aug 10 06:27:56 dist-test-slave-45dx krb5kdc[27717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343275, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.3@KRBTEST.COM
I20260810 06:27:56.925385 30443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:56.910961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:32801 (local address 127.4.73.1:33985)
0810 06:27:56.911399 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:56.911413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:56.911542 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:56.913573 (+ 2031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:56.913579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:56.914206 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:56.914718 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:56.914733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.915971 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.915976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:56.916888 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:56.916899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.917600 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.917603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:56.917816 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:56.918479 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:56.918501 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:56.920156 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:56.922775 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:56.922779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:56.922789 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:56.923078 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:56.923445 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:56.923451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:56.923454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:56.923603 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:56.924429 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:56.924431 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:56.924626 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 06:27:56.924827 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 06:27:56.925183 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":309,"spinlock_wait_cycles":19456,"thread_start_us":97,"threads_started":1}
I20260810 06:27:56.925384 30435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:56.910961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43905 (local address 127.4.73.1:48561)
0810 06:27:56.911360 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:56.911377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:27:56.911500 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:27:56.913572 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:27:56.913578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:27:56.914216 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:27:56.914719 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:56.914733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.915963 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.915971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:56.916878 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:27:56.916888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.917578 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.917582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:27:56.917836 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:56.918545 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:27:56.918557 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:27:56.920529 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:27:56.922826 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:56.922829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:56.922830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:56.923133 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:56.923510 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:27:56.923512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:27:56.923513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:27:56.923606 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:27:56.924345 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:27:56.924351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:27:56.924626 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 06:27:56.924827 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 06:27:56.925235 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":246,"spinlock_wait_cycles":6144}
I20260810 06:27:56.925437 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:56.911188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:33985 (local address 127.4.73.2:32801)
0810 06:27:56.911621 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:56.911649 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:56.911693 (+ 44us) server_negotiation.cc:407] Connection header received
0810 06:27:56.912669 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:56.912679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:56.913120 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:56.913479 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:56.914910 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.915837 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.917017 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.917469 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.917644 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:56.920437 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:56.920441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:56.920453 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:56.920481 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:56.922659 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:56.923193 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:56.923202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:56.923207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:56.923338 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:56.923705 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:56.923709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:56.923711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:56.924210 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:56.924385 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:56.924976 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 06:27:56.925255 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":124,"threads_started":1}
I20260810 06:27:56.926095 30445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:56.911194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:48561 (local address 127.4.73.3:43905)
0810 06:27:56.911621 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 06:27:56.911650 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 06:27:56.911693 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:27:56.912670 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:27:56.912678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:27:56.913146 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:27:56.914179 (+ 1033us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:27:56.914886 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.915814 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.917004 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:27:56.917448 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:27:56.917623 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:27:56.920722 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:27:56.920730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:27:56.920746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:27:56.920783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:27:56.922659 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:56.923250 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:56.923255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:56.923259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:56.923374 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:27:56.923716 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:27:56.923719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:27:56.923721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:27:56.924160 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:27:56.924349 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 06:27:56.924976 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 06:27:56.925255 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":114,"threads_started":1}
W20260810 06:27:56.927955 29794 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
W20260810 06:27:56.928134 29793 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
I20260810 06:27:56.928246 29793 leader_election.cc:302] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a09c407511b04fe9a9450d31f14a0616; no voters: 143c4969c21f42a48e6678feff8cb44e, 91dea911ddb34e3a99387017780da2cd
I20260810 06:27:56.928577 30439 raft_consensus.cc:2726] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:27:57.513322 30450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:57.512927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:50749)
0810 06:27:57.513169 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:57.513224 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":80,"threads_started":1}
I20260810 06:27:57.636531 30451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:57.636065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:48151)
0810 06:27:57.636385 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:57.636441 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260810 06:27:57.816268 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:57.815851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:45199)
0810 06:27:57.816134 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:57.816185 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
I20260810 06:27:58.514482 30454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:58.514058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:57707)
0810 06:27:58.514324 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:58.514382 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260810 06:27:58.637619 30456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:58.637241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:52939)
0810 06:27:58.637484 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:58.637534 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260810 06:27:58.718699 30459 raft_consensus.cc:482] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:27:58.718860 30459 raft_consensus.cc:504] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:27:58.719252 30459 leader_election.cc:288] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
W20260810 06:27:58.720171 29793 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
W20260810 06:27:58.720355 29794 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
I20260810 06:27:58.720449 29794 leader_election.cc:302] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a09c407511b04fe9a9450d31f14a0616; no voters: 143c4969c21f42a48e6678feff8cb44e, 91dea911ddb34e3a99387017780da2cd
I20260810 06:27:58.720656 30459 raft_consensus.cc:2726] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:27:58.817344 30461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:58.816928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:45669)
0810 06:27:58.817184 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:58.817242 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260810 06:27:59.515631 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:59.515195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:36615)
0810 06:27:59.515480 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:59.515537 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20260810 06:27:59.638851 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:59.638486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:55465)
0810 06:27:59.638720 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:59.638769 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260810 06:27:59.818415 30468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:27:59.818031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:44503)
0810 06:27:59.818275 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 06:27:59.818325 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260810 06:28:00.517207 30469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:00.516645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:47677)
0810 06:28:00.517002 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:00.517079 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":75,"threads_started":1}
I20260810 06:28:00.640021 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:00.639635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:51327)
0810 06:28:00.639869 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:00.639922 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260810 06:28:00.819625 30474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:00.819129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:47731)
0810 06:28:00.819478 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:00.819536 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":94,"threads_started":1}
I20260810 06:28:01.342092 30480 raft_consensus.cc:482] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:28:01.342267 30480 raft_consensus.cc:504] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:28:01.342625 30480 leader_election.cc:288] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
W20260810 06:28:01.343416 29793 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
W20260810 06:28:01.343571 29794 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
I20260810 06:28:01.343657 29794 leader_election.cc:302] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a09c407511b04fe9a9450d31f14a0616; no voters: 143c4969c21f42a48e6678feff8cb44e, 91dea911ddb34e3a99387017780da2cd
I20260810 06:28:01.343806 30480 raft_consensus.cc:2726] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:28:01.518426 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:01.518017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:47303)
0810 06:28:01.518260 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:01.518328 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260810 06:28:01.641227 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:01.640825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:43063)
0810 06:28:01.641081 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:01.641137 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260810 06:28:01.820995 30483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:01.820560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:57559)
0810 06:28:01.820831 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:01.820888 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":96,"threads_started":1}
I20260810 06:28:02.519646 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:02.519227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:40225)
0810 06:28:02.519494 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:02.519548 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
I20260810 06:28:02.642488 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:02.641990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:59463)
0810 06:28:02.642312 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:02.642376 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":77,"threads_started":1}
I20260810 06:28:02.822494 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:02.821970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:37227)
0810 06:28:02.822329 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:02.822390 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":71,"threads_started":1}
I20260810 06:28:03.520844 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:03.520417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:44231)
0810 06:28:03.520684 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:03.520740 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
I20260810 06:28:03.643689 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:03.643253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:57321)
0810 06:28:03.643535 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:03.643595 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
I20260810 06:28:03.823668 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:03.823248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:49059)
0810 06:28:03.823511 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:03.823568 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260810 06:28:04.522055 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:04.521618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:52689)
0810 06:28:04.521900 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:04.521957 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260810 06:28:04.645030 30492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:04.644632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:55807)
0810 06:28:04.644880 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:04.644936 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260810 06:28:04.693526 30496 raft_consensus.cc:482] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:28:04.693665 30496 raft_consensus.cc:504] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91dea911ddb34e3a99387017780da2cd" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 32801 } } peers { permanent_uuid: "143c4969c21f42a48e6678feff8cb44e" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43905 } } peers { permanent_uuid: "a09c407511b04fe9a9450d31f14a0616" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 37679 } }
I20260810 06:28:04.694118 30496 leader_election.cc:288] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801), 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905)
W20260810 06:28:04.694916 29794 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 91dea911ddb34e3a99387017780da2cd (127.4.73.2:32801): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
W20260810 06:28:04.695030 29793 leader_election.cc:341] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 143c4969c21f42a48e6678feff8cb44e (127.4.73.3:43905): Not found: Tablet not found: e3729f8a982342928952a348704ee3df
I20260810 06:28:04.695132 29793 leader_election.cc:302] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a09c407511b04fe9a9450d31f14a0616; no voters: 143c4969c21f42a48e6678feff8cb44e, 91dea911ddb34e3a99387017780da2cd
I20260810 06:28:04.695304 30496 raft_consensus.cc:2726] T e3729f8a982342928952a348704ee3df P a09c407511b04fe9a9450d31f14a0616 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:28:04.824870 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:04.824445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:48911)
0810 06:28:04.824702 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:04.824780 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":94,"threads_started":1}
I20260810 06:28:05.523561 30501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:05.523039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:48625)
0810 06:28:05.523328 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:05.523408 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260810 06:28:05.646237 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:05.645815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:57895)
0810 06:28:05.646077 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:05.646136 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260810 06:28:05.826051 30504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:05.825609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:51237)
0810 06:28:05.825890 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:05.825947 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260810 06:28:06.525043 30505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:06.524468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:56011)
0810 06:28:06.524864 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:06.524943 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260810 06:28:06.647389 30506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:06.646945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:40651)
0810 06:28:06.647225 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:06.647287 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260810 06:28:06.827234 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:06.826784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:33797)
0810 06:28:06.827068 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:06.827128 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260810 06:28:07.526100 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:07.525687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:50963)
0810 06:28:07.525952 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:07.526007 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20260810 06:28:07.648545 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:07.648082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:51979)
0810 06:28:07.648397 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:07.648451 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":121,"threads_started":1}
I20260810 06:28:07.828475 30513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:07.827960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:51639)
0810 06:28:07.828260 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:07.828374 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260810 06:28:08.527240 30516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:08.526779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:51929)
0810 06:28:08.527084 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:08.527140 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260810 06:28:08.649751 30517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:08.649320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:58925)
0810 06:28:08.649609 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:08.649662 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
I20260810 06:28:08.829604 30519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:08.829213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:44103)
0810 06:28:08.829452 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:08.829514 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260810 06:28:09.528466 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:09.528041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:56983)
0810 06:28:09.528282 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:09.528354 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260810 06:28:09.650931 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:09.650501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:41415)
0810 06:28:09.650782 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:09.650838 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260810 06:28:09.830673 30528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:09.830262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:49627)
0810 06:28:09.830527 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:09.830580 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260810 06:28:10.529846 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:10.529371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.1:42165)
0810 06:28:10.529664 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:10.529722 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260810 06:28:10.652483 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:10.651959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.2:51469)
0810 06:28:10.652256 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:10.652356 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20260810 06:28:10.824121 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 29698
I20260810 06:28:10.833917 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:10.833361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43645 (local address 127.4.73.3:55393)
0810 06:28:10.833707 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:10.833777 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:43645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":76,"threads_started":1}
I20260810 06:28:10.840554 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 29988
I20260810 06:28:10.854074 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 30210
2026-08-10T06:28:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19759 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:28:10 dist-test-slave-45dx krb5kdc[30539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:28:10 dist-test-slave-45dx krb5kdc[30539](info): set up 2 sockets
Aug 10 06:28:10 dist-test-slave-45dx krb5kdc[30539](info): commencing operation
krb5kdc: starting...
W20260810 06:28:12.169358 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:28:12 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:28:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:28:12.297190 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44121
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:44121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:28:12.407176 30560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:12.407553 30560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:12.407617 30560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:12.411999 30560 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 06:28:12.412063 30560 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 06:28:12.412079 30560 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 06:28:12.412093 30560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:28:12.412111 30560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:12.412129 30560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:12.412144 30560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:28:12.412159 30560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:28:12.417559 30560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:44121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:12.418488 30560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:12.419447 30560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:12.428292 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:12.428472 30560 server_base.cc:1034] running on GCE node
W20260810 06:28:12.428282 30567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:12.428284 30568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:12.429240 30560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:12.430282 30560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:12.431469 30560 hybrid_clock.cc:648] HybridClock initialized: now 1786343292431440 us; error 41 us; skew 500 ppm
Aug 10 06:28:12 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343292, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:12.438591 30560 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:28:12.439509 30560 webserver.cc:460] Webserver started at http://127.4.73.62:38603/ using document root <none> and password file <none>
I20260810 06:28:12.440057 30560 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:12.440148 30560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:12.440433 30560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:28:12.442435 30560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "6717415825db4592b97290e57aae85fd"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "16ff874c78a5109b253bce142fa458a3"
server_key_iv: "aca4a9caf644e6671dc2feb3cb228452"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.443070 30560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "6717415825db4592b97290e57aae85fd"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "16ff874c78a5109b253bce142fa458a3"
server_key_iv: "aca4a9caf644e6671dc2feb3cb228452"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.446565 30560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 06:28:12.448911 30581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:12.450026 30560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:28:12.450165 30560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "6717415825db4592b97290e57aae85fd"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "16ff874c78a5109b253bce142fa458a3"
server_key_iv: "aca4a9caf644e6671dc2feb3cb228452"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.450268 30560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:12.466480 30560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:12.469579 30560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:12.469782 30560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:12.477021 30560 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:44121
I20260810 06:28:12.477303 30670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:44121 every 8 connection(s)
I20260810 06:28:12.477998 30560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:28:12.480778 30673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:28:12.483867 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 30560
I20260810 06:28:12.484031 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:28:12.484364 4388 external_mini_cluster.cc:1428] Setting key 3cd5ad66528f3ab10f11e43e058e7289
I20260810 06:28:12.486902 30673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Bootstrap starting.
I20260810 06:28:12.489638 30673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Neither blocks nor log segments found. Creating new log.
I20260810 06:28:12.490442 30673 log.cc:824] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:12.492674 30673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: No bootstrap required, opened a new log
Aug 10 06:28:12 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:28:12.496173 30673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:12.496464 30673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:28:12.496543 30673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6717415825db4592b97290e57aae85fd, State: Initialized, Role: FOLLOWER
I20260810 06:28:12.496971 30673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:12.497117 30673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:28:12.497184 30673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:28:12.497304 30673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:28:12.498202 30673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:12.498560 30673 leader_election.cc:302] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6717415825db4592b97290e57aae85fd; no voters:
I20260810 06:28:12.498930 30673 leader_election.cc:288] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:28:12.499168 30685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:28:12.499404 30685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 LEADER]: Becoming Leader. State: Replica: 6717415825db4592b97290e57aae85fd, State: Running, Role: LEADER
I20260810 06:28:12.499719 30685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:12.500066 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:12.485754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.4.73.62:44121)
0810 06:28:12.486190 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:12.486207 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:12.486245 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:28:12.486901 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:12.486918 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:12.487330 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:12.487632 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:12.488674 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:12.489666 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:12.490576 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:12.491236 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:12.491442 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:12.494320 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:12.494325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:12.494337 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:12.494364 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:12.496871 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:12.497394 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:12.497402 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:12.497410 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:12.497550 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:12.497857 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:12.497860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:12.497862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:12.498252 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:12.498491 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:12.498770 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 06:28:12.499098 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":145,"threads_started":1}
I20260810 06:28:12.500237 30673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:28:12.501389 30685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6717415825db4592b97290e57aae85fd. Latest consensus state: current_term: 1 leader_uuid: "6717415825db4592b97290e57aae85fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } } }
I20260810 06:28:12.501503 30685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: This master's current role is: LEADER
I20260810 06:28:12.501650 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6717415825db4592b97290e57aae85fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } } }
I20260810 06:28:12.501760 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: This master's current role is: LEADER
W20260810 06:28:12.505492 30706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:28:12.505586 30706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:28:12.505666 30696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:28:12.506371 30696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:28:12.511633 30696 catalog_manager.cc:1347] Generated new cluster ID: bcc839e0182f43c1983e695cc79ae31b
I20260810 06:28:12.511696 30696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:28:12.539201 30696 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:28:12.540531 30696 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:28:12.546101 30696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Generated new TSK 0
I20260810 06:28:12.547165 30696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:28:12.584007 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44121
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:28:12.699249 30721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:12.699512 30721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:12.699589 30721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:12.704028 30721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:12.704087 30721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:12.704190 30721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:28:12.709362 30721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:12.710397 30721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:12.711429 30721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:12.721127 30737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:12.721073 30731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:12.721084 30734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:12.721675 30721 server_base.cc:1034] running on GCE node
I20260810 06:28:12.722126 30721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:12.722728 30721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:12.723906 30721 hybrid_clock.cc:648] HybridClock initialized: now 1786343292723876 us; error 41 us; skew 500 ppm
Aug 10 06:28:12 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343292, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:12.726677 30721 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:28:12.727527 30721 webserver.cc:460] Webserver started at http://127.4.73.1:40215/ using document root <none> and password file <none>
I20260810 06:28:12.728088 30721 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:12.728186 30721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:12.728475 30721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:28:12.730533 30721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "11f48fb346c74bf6980b680bb874785c"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "f76fe948902111d61332901fb6c909ee"
server_key_iv: "6876258f40f79573bda1734f8145196c"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.731119 30721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "11f48fb346c74bf6980b680bb874785c"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "f76fe948902111d61332901fb6c909ee"
server_key_iv: "6876258f40f79573bda1734f8145196c"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.734750 30721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:28:12.737140 30747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:12.738411 30721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 06:28:12.738543 30721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "11f48fb346c74bf6980b680bb874785c"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "f76fe948902111d61332901fb6c909ee"
server_key_iv: "6876258f40f79573bda1734f8145196c"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.738654 30721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:12.763599 30721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:12.766942 30721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:12.767171 30721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:12.767797 30721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:12.768684 30721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:28:12.768759 30721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:12.768837 30721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:28:12.768882 30721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:12.777861 30721 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:40933
I20260810 06:28:12.777885 30938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:40933 every 8 connection(s)
I20260810 06:28:12.778741 30721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:28:12.781778 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 30721
I20260810 06:28:12.781960 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:28:12.782295 4388 external_mini_cluster.cc:1428] Setting key dd45c362ba0b3bfc3918ba359ce323c4
Aug 10 06:28:12 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343292, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:28:12.793453 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:12.780555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:38213 (local address 127.4.73.62:44121)
0810 06:28:12.780695 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:12.780698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:12.781766 (+ 1068us) server_negotiation.cc:407] Connection header received
0810 06:28:12.782818 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:12.782822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:12.782880 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:12.782964 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:12.784109 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:12.784732 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:12.786177 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:12.786413 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:12.786496 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:12.789487 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:12.789494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:12.789500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:12.789536 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:12.791290 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:12.791785 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:12.791791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:12.791793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:12.791912 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:12.792256 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:12.792260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:12.792262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:12.792540 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:12.792657 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:12.793145 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 06:28:12.793282 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:28:12.794209 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:12.780961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:38213)
0810 06:28:12.781595 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:12.781636 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:12.782612 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:12.783130 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:12.783137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:12.783485 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:12.783956 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:12.783968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:12.784873 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:12.784881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:12.786030 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:12.786046 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:12.786577 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:12.786581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:12.786797 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:12.787517 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:12.787534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:12.789297 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:12.791411 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:12.791415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:12.791426 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:12.791679 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:12.792013 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:12.792015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:12.792016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:12.792155 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:12.792645 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:12.792651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:12.792885 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 06:28:12.793040 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 06:28:12.793292 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":112,"threads_started":1}
I20260810 06:28:12.795596 30939 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44121
I20260810 06:28:12.795840 30939 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:28:12.796433 30939 heartbeater.cc:511] Master 127.4.73.62:44121 requested a full tablet report, sending...
I20260810 06:28:12.798049 30610 ts_manager.cc:194] Registered new tserver with Master: 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933)
I20260810 06:28:12.799700 30610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:38213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:28:12.813690 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44121
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:28:12.929642 30951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:12.929935 30951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:12.930009 30951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:12.934590 30951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:12.934676 30951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:12.934777 30951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:28:12.940210 30951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:12.941336 30951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:12.942481 30951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:12.952270 30964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:12.952270 30962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:12.952325 30969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:12.952639 30951 server_base.cc:1034] running on GCE node
I20260810 06:28:12.953063 30951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:12.953712 30951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:12.954895 30951 hybrid_clock.cc:648] HybridClock initialized: now 1786343292954857 us; error 46 us; skew 500 ppm
Aug 10 06:28:12 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343292, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:12.957860 30951 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:28:12.958669 30951 webserver.cc:460] Webserver started at http://127.4.73.2:37083/ using document root <none> and password file <none>
I20260810 06:28:12.959188 30951 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:12.959249 30951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:12.959410 30951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:28:12.961493 30951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "d55186e037a947418cf0b19a5d455570"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "92a5151ef7f809c5462c17c973c47981"
server_key_iv: "5052e9d1def7db36453ec568d57c428e"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.962023 30951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "d55186e037a947418cf0b19a5d455570"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "92a5151ef7f809c5462c17c973c47981"
server_key_iv: "5052e9d1def7db36453ec568d57c428e"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.965603 30951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:28:12.967854 30980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:12.969053 30951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:28:12.969149 30951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "d55186e037a947418cf0b19a5d455570"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "92a5151ef7f809c5462c17c973c47981"
server_key_iv: "5052e9d1def7db36453ec568d57c428e"
server_key_version: "encryptionkey@0"
I20260810 06:28:12.969233 30951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:12.982003 30951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:12.984802 30951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:12.984961 30951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:12.985469 30951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:12.986220 30951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:28:12.986267 30951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:12.986306 30951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:28:12.986330 30951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:12.995059 30951 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:44155
I20260810 06:28:12.995081 31177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:44155 every 8 connection(s)
I20260810 06:28:12.995922 30951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:28:13.000624 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 30951
I20260810 06:28:13.000748 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:28:13.001101 4388 external_mini_cluster.cc:1428] Setting key b88f3f34ddd223ef6c063de359ee53ab
Aug 10 06:28:13 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343292, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:28:13.011309 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:12.997728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:58843 (local address 127.4.73.62:44121)
0810 06:28:12.997867 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:12.997874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:12.999134 (+ 1260us) server_negotiation.cc:407] Connection header received
0810 06:28:13.000189 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.000193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.000242 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.000390 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:13.001982 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.002719 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.004098 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.004404 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.004526 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.007206 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:13.007209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:13.007211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:13.007237 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:13.008978 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:13.009633 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:13.009635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:13.009637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:13.009714 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:13.010152 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:13.010154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:13.010155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:13.010330 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:13.010408 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.011083 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.011185 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:28:13.012589 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:12.998065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:58843)
0810 06:28:12.998919 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:12.998959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:12.999972 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.000593 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.000603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.001109 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:13.001835 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.001857 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.002926 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.002928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.003961 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.003971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.004570 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.004574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.004744 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.005386 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:13.005403 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:13.007033 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:13.009102 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:13.009110 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:13.009129 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:13.009527 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:13.009821 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:13.009826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:13.009831 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:13.010047 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:13.010449 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:13.010458 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:13.010806 (+ 348us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.011040 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.011422 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":526,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:28:13.013809 31180 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44121
I20260810 06:28:13.014053 31180 heartbeater.cc:464] Registering TS with master...
I20260810 06:28:13.014569 31180 heartbeater.cc:511] Master 127.4.73.62:44121 requested a full tablet report, sending...
I20260810 06:28:13.015722 30610 ts_manager.cc:194] Registered new tserver with Master: d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:28:13.016459 30610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:58843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:28:13.031858 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44121
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:28:13.148659 31188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:13.148936 31188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:13.148981 31188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:13.153023 31188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:13.153100 31188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:13.153158 31188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:28:13.158763 31188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:13.159799 31188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:13.160838 31188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:13.170920 31199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:13.170985 31188 server_base.cc:1034] running on GCE node
W20260810 06:28:13.170949 31195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:13.171077 31196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:13.171626 31188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:13.172273 31188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:13.173520 31188 hybrid_clock.cc:648] HybridClock initialized: now 1786343293173479 us; error 44 us; skew 500 ppm
Aug 10 06:28:13 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343293, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:13.176419 31188 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:28:13.177289 31188 webserver.cc:460] Webserver started at http://127.4.73.3:46403/ using document root <none> and password file <none>
I20260810 06:28:13.177888 31188 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:13.177989 31188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:13.178233 31188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:28:13.180409 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "d0e5ae51cabe47dea4e583e3c30ec627"
format_stamp: "Formatted at 2026-08-10 06:28:13 on dist-test-slave-45dx"
server_key: "14f6bf36a40f51e10393c442bf490146"
server_key_iv: "33369568bbe61fcbde87c6e99a7ae7f6"
server_key_version: "encryptionkey@0"
I20260810 06:28:13.181018 31188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "d0e5ae51cabe47dea4e583e3c30ec627"
format_stamp: "Formatted at 2026-08-10 06:28:13 on dist-test-slave-45dx"
server_key: "14f6bf36a40f51e10393c442bf490146"
server_key_iv: "33369568bbe61fcbde87c6e99a7ae7f6"
server_key_version: "encryptionkey@0"
I20260810 06:28:13.185128 31188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 06:28:13.187640 31212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:13.189159 31188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:28:13.189314 31188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "d0e5ae51cabe47dea4e583e3c30ec627"
format_stamp: "Formatted at 2026-08-10 06:28:13 on dist-test-slave-45dx"
server_key: "14f6bf36a40f51e10393c442bf490146"
server_key_iv: "33369568bbe61fcbde87c6e99a7ae7f6"
server_key_version: "encryptionkey@0"
I20260810 06:28:13.189435 31188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:13.205114 31188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:13.208603 31188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:13.208822 31188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:13.209508 31188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:13.210357 31188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:28:13.210430 31188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:13.210503 31188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:28:13.210554 31188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:13.219379 31188 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:38029
I20260810 06:28:13.219406 31421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:38029 every 8 connection(s)
I20260810 06:28:13.220250 31188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
Aug 10 06:28:13 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343293, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:28:13.229568 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 31188
I20260810 06:28:13.229717 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:28:13.230057 4388 external_mini_cluster.cc:1428] Setting key 3edc951c8e257bcb29b9ee6895632b6c
I20260810 06:28:13.234125 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.222096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:39723 (local address 127.4.73.62:44121)
0810 06:28:13.222234 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.222238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.223232 (+ 994us) server_negotiation.cc:407] Connection header received
0810 06:28:13.224137 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.224140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.224201 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.224282 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:13.225465 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.226014 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.227144 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.227400 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.227510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.230303 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:13.230308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:13.230311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:13.230353 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:13.232137 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:13.232738 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:13.232741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:13.232742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:13.232854 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:13.233163 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:13.233165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:13.233166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:13.233352 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:13.233428 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.233905 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.234006 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:28:13.234898 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.222446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:39723)
0810 06:28:13.223086 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.223127 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.223932 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.224475 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.224482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.224832 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:13.225314 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.225328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.226163 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.226168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.227015 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.227024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.227562 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.227567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.227728 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.228411 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:13.228427 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:13.230112 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:13.232293 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:13.232320 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:13.232336 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:13.232645 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:13.232941 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:13.232943 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:13.232945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:13.233078 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:13.233439 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:13.233444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:13.233672 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.233840 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.234093 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":111,"threads_started":1}
I20260810 06:28:13.236032 31422 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44121
I20260810 06:28:13.236276 31422 heartbeater.cc:464] Registering TS with master...
I20260810 06:28:13.236840 31422 heartbeater.cc:511] Master 127.4.73.62:44121 requested a full tablet report, sending...
I20260810 06:28:13.237851 30610 ts_manager.cc:194] Registered new tserver with Master: d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029)
I20260810 06:28:13.238427 30610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:39723
I20260810 06:28:13.244236 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:28:13.253389 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.246383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.4.73.62:44121)
0810 06:28:13.246524 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.246530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.246675 (+ 145us) server_negotiation.cc:407] Connection header received
0810 06:28:13.246787 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.246791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.246841 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.246928 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:13.247585 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.248097 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.248996 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.249218 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.249289 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.250604 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:13.250607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:13.250608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:13.250625 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:13.251854 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:13.252232 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:13.252234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:13.252235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:13.252374 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:13.252634 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:13.252636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:13.252637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:13.252838 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:13.252940 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.253131 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.253220 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:28:13.256515 30610 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:28:13.259462 30610 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:28:13.273761 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.267947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:44155 (local address 127.0.0.1:45506)
0810 06:28:13.268983 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.268997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.269090 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.269747 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.269750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.269783 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:13.269863 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.269867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.270837 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.270840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.272196 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.272206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.273203 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.273207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.273311 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.273360 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.273475 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.273610 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":161,"spinlock_wait_cycles":5504,"thread_start_us":100,"threads_started":1}
I20260810 06:28:13.273763 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.268456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.4.73.2:44155)
0810 06:28:13.269000 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.269013 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.269076 (+ 63us) server_negotiation.cc:407] Connection header received
0810 06:28:13.269230 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.269236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.269425 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.269578 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:13.269987 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.270696 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.272338 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.273042 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.273225 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.273373 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.273517 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.273612 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":103,"threads_started":1}
I20260810 06:28:13.274201 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.267679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40933 (local address 127.0.0.1:38442)
0810 06:28:13.268826 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.268841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.268946 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.269419 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.269425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.269447 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:13.269599 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.269608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.271052 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.271056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.272596 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.272607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.273806 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.273811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.273898 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.273924 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.273999 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.274067 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"mutex_wait_us":155,"thread_start_us":101,"threads_started":1}
I20260810 06:28:13.274708 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.268421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.4.73.1:40933)
0810 06:28:13.268814 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.268833 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.268945 (+ 112us) server_negotiation.cc:407] Connection header received
0810 06:28:13.269060 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.269066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.269268 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.269432 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:13.269720 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.270907 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.272751 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.273528 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.273617 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.273742 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.274463 (+ 721us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.274562 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":115,"threads_started":1}
I20260810 06:28:13.274797 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.267619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:38029 (local address 127.0.0.1:58224)
0810 06:28:13.268465 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.268520 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.268647 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.269753 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.269756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.269790 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:13.269887 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.269893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.271217 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.271220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.272598 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.272607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.274215 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.274219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.274516 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.274538 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.274600 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.274660 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":135,"spinlock_wait_cycles":1408,"thread_start_us":75,"threads_started":1}
I20260810 06:28:13.275220 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.267736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.4.73.3:38029)
0810 06:28:13.269231 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.269242 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.269262 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:28:13.269326 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.269330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.269472 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.269600 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:13.270021 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.271081 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.272737 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.273584 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.273683 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.273810 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.275009 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.275101 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1373,"thread_start_us":155,"threads_started":1}
I20260810 06:28:13.277166 31067 tablet_service.cc:1467] Processing CreateTablet for tablet 931fe2191b9c4d118b773a1cfb6aebbb (DEFAULT_TABLE table=test-table [id=92947f51ca0646978e784773832337eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:28:13.277896 31292 tablet_service.cc:1467] Processing CreateTablet for tablet 931fe2191b9c4d118b773a1cfb6aebbb (DEFAULT_TABLE table=test-table [id=92947f51ca0646978e784773832337eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:28:13.278344 31067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931fe2191b9c4d118b773a1cfb6aebbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:28:13.278915 31292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931fe2191b9c4d118b773a1cfb6aebbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:28:13.278640 30829 tablet_service.cc:1467] Processing CreateTablet for tablet 931fe2191b9c4d118b773a1cfb6aebbb (DEFAULT_TABLE table=test-table [id=92947f51ca0646978e784773832337eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:28:13.279428 30829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 931fe2191b9c4d118b773a1cfb6aebbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:28:13.285259 31447 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Bootstrap starting.
I20260810 06:28:13.286473 31451 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Bootstrap starting.
I20260810 06:28:13.287817 31447 tablet_bootstrap.cc:654] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Neither blocks nor log segments found. Creating new log.
I20260810 06:28:13.288592 31451 tablet_bootstrap.cc:654] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Neither blocks nor log segments found. Creating new log.
I20260810 06:28:13.289261 31447 log.cc:824] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:13.289814 31451 log.cc:824] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:13.289997 31450 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Bootstrap starting.
I20260810 06:28:13.291152 31447 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: No bootstrap required, opened a new log
I20260810 06:28:13.291368 31447 ts_tablet_manager.cc:1397] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:28:13.291740 31451 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: No bootstrap required, opened a new log
I20260810 06:28:13.291929 31451 ts_tablet_manager.cc:1397] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:28:13.292506 31450 tablet_bootstrap.cc:654] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Neither blocks nor log segments found. Creating new log.
I20260810 06:28:13.293460 31450 log.cc:824] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:13.294844 31447 raft_consensus.cc:348] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.295140 31447 raft_consensus.cc:374] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:28:13.295203 31447 raft_consensus.cc:729] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f48fb346c74bf6980b680bb874785c, State: Initialized, Role: FOLLOWER
I20260810 06:28:13.295526 31450 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: No bootstrap required, opened a new log
I20260810 06:28:13.295698 31450 ts_tablet_manager.cc:1397] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 06:28:13.295696 31447 consensus_queue.cc:260] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.295890 31451 raft_consensus.cc:348] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.296150 31451 raft_consensus.cc:374] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:28:13.296211 31451 raft_consensus.cc:729] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e5ae51cabe47dea4e583e3c30ec627, State: Initialized, Role: FOLLOWER
I20260810 06:28:13.296501 30939 heartbeater.cc:503] Master 127.4.73.62:44121 was elected leader, sending a full tablet report...
I20260810 06:28:13.296754 31447 ts_tablet_manager.cc:1428] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:28:13.296711 31451 consensus_queue.cc:260] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.297345 31422 heartbeater.cc:503] Master 127.4.73.62:44121 was elected leader, sending a full tablet report...
I20260810 06:28:13.297617 31451 ts_tablet_manager.cc:1428] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:28:13.299271 31450 raft_consensus.cc:348] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.299501 31450 raft_consensus.cc:374] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:28:13.299607 31450 raft_consensus.cc:729] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55186e037a947418cf0b19a5d455570, State: Initialized, Role: FOLLOWER
I20260810 06:28:13.300125 31450 consensus_queue.cc:260] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.300973 31180 heartbeater.cc:503] Master 127.4.73.62:44121 was elected leader, sending a full tablet report...
I20260810 06:28:13.301146 31450 ts_tablet_manager.cc:1428] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260810 06:28:13.471241 31423 tablet.cc:2369] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:28:13.497155 31181 tablet.cc:2369] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:28:13.530114 30941 tablet.cc:2369] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:28:13.588629 31466 raft_consensus.cc:482] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:28:13.589011 31462 raft_consensus.cc:482] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:28:13.589008 31466 raft_consensus.cc:504] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.589224 31462 raft_consensus.cc:504] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.590318 31466 leader_election.cc:288] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933)
I20260810 06:28:13.591445 31462 leader_election.cc:288] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
I20260810 06:28:13.595649 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.590938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:46983 (local address 127.4.73.2:44155)
0810 06:28:13.591059 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.591065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.591151 (+ 86us) server_negotiation.cc:407] Connection header received
0810 06:28:13.591374 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.591379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.591449 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.591499 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:13.591957 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.592760 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.594393 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.594960 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595030 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.595090 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.595439 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.595517 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 06:28:13.595911 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.591108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:40471 (local address 127.4.73.1:40933)
0810 06:28:13.591207 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.591211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.591883 (+ 672us) server_negotiation.cc:407] Connection header received
0810 06:28:13.592053 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.592056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.592107 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.592151 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:13.592542 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.593234 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.594728 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.595268 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595330 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.595399 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.595715 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.595789 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":1408}
I20260810 06:28:13.596166 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.590557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:44155 (local address 127.4.73.1:46983)
0810 06:28:13.591054 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.591072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.591239 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.591657 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.591663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.591682 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:13.591816 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.591826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.592929 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.592932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.594261 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.594273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.595180 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.595289 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.595321 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.595841 (+ 520us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.595901 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":156,"threads_started":1}
I20260810 06:28:13.596238 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.590987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:48161 (local address 127.4.73.3:38029)
0810 06:28:13.591351 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.591358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.591381 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:28:13.591443 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.591446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.591499 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.591577 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:13.592015 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.592906 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.594770 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.595471 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595553 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.595620 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.596003 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.596099 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290}
I20260810 06:28:13.596490 31476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.591087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:45607 (local address 127.4.73.3:38029)
0810 06:28:13.591714 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.591724 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.591735 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:28:13.591784 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.591787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.591831 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.591923 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:13.592356 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.593265 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.594993 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.595630 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595728 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.595798 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.596250 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.596344 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":104,"threads_started":1}
I20260810 06:28:13.596632 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.590704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40933 (local address 127.4.73.2:40471)
0810 06:28:13.591799 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.591813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.591945 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.592265 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.592268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.592283 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:13.592421 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.592427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.593375 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.593378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.594606 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.594615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.595498 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.595578 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.595597 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.596232 (+ 635us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.596486 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":32,"thread_start_us":90,"threads_started":1}
I20260810 06:28:13.596786 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.590557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:38029 (local address 127.4.73.1:45607)
0810 06:28:13.591366 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.591379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.591477 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.592049 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.592053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.592072 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:13.592188 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.592195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.593399 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.593403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.594856 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.594867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.595864 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.595993 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.596018 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.596086 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.596150 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":84,"spinlock_wait_cycles":1792,"thread_start_us":105,"threads_started":1}
I20260810 06:28:13.596985 31100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "11f48fb346c74bf6980b680bb874785c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55186e037a947418cf0b19a5d455570" is_pre_election: true
I20260810 06:28:13.597246 31100 raft_consensus.cc:2454] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f48fb346c74bf6980b680bb874785c in term 0.
I20260810 06:28:13.597370 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "d55186e037a947418cf0b19a5d455570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f48fb346c74bf6980b680bb874785c" is_pre_election: true
I20260810 06:28:13.597535 31467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.590557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:38029 (local address 127.4.73.2:48161)
0810 06:28:13.591200 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.591218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:13.591331 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:13.591708 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:13.591712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:13.591734 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:13.591873 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.591884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.593056 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.593059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.594624 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:13.594635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.595729 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.595732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:13.595852 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.595875 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:28:13.596470 (+ 595us) client_negotiation.cc:240] Negotiation successful
0810 06:28:13.596630 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":66,"thread_start_us":86,"threads_started":1}
I20260810 06:28:13.597563 30866 raft_consensus.cc:2454] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55186e037a947418cf0b19a5d455570 in term 0.
I20260810 06:28:13.597610 31330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "11f48fb346c74bf6980b680bb874785c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" is_pre_election: true
I20260810 06:28:13.597764 30750 leader_election.cc:302] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c, d55186e037a947418cf0b19a5d455570; no voters:
I20260810 06:28:13.597900 31330 raft_consensus.cc:2454] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11f48fb346c74bf6980b680bb874785c in term 0.
I20260810 06:28:13.598062 30981 leader_election.cc:302] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c, d55186e037a947418cf0b19a5d455570; no voters:
I20260810 06:28:13.598091 31462 raft_consensus.cc:2781] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:28:13.598182 31462 raft_consensus.cc:482] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:28:13.598242 31462 raft_consensus.cc:3037] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:28:13.598206 31328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "d55186e037a947418cf0b19a5d455570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" is_pre_election: true
I20260810 06:28:13.598335 31466 raft_consensus.cc:2781] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:28:13.598340 31328 raft_consensus.cc:2454] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d55186e037a947418cf0b19a5d455570 in term 0.
I20260810 06:28:13.598399 31466 raft_consensus.cc:482] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:28:13.598450 31466 raft_consensus.cc:3037] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:28:13.599532 31466 raft_consensus.cc:504] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.599538 31462 raft_consensus.cc:504] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.599901 31466 leader_election.cc:288] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [CANDIDATE]: Term 1 election: Requested vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933)
I20260810 06:28:13.600116 31462 leader_election.cc:288] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 1 election: Requested vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
I20260810 06:28:13.600266 31328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "d55186e037a947418cf0b19a5d455570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e5ae51cabe47dea4e583e3c30ec627"
I20260810 06:28:13.600329 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "d55186e037a947418cf0b19a5d455570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f48fb346c74bf6980b680bb874785c"
I20260810 06:28:13.600368 31330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "11f48fb346c74bf6980b680bb874785c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e5ae51cabe47dea4e583e3c30ec627"
I20260810 06:28:13.600416 31328 raft_consensus.cc:3037] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:28:13.600517 30866 raft_consensus.cc:2379] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d55186e037a947418cf0b19a5d455570 in current term 1: Already voted for candidate 11f48fb346c74bf6980b680bb874785c in this term.
I20260810 06:28:13.601058 31100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "931fe2191b9c4d118b773a1cfb6aebbb" candidate_uuid: "11f48fb346c74bf6980b680bb874785c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55186e037a947418cf0b19a5d455570"
I20260810 06:28:13.601234 31100 raft_consensus.cc:2379] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 11f48fb346c74bf6980b680bb874785c in current term 1: Already voted for candidate d55186e037a947418cf0b19a5d455570 in this term.
I20260810 06:28:13.601665 30750 leader_election.cc:302] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c; no voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570
I20260810 06:28:13.601856 31462 raft_consensus.cc:2726] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 06:28:13.602087 31328 raft_consensus.cc:2454] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d55186e037a947418cf0b19a5d455570 in term 1.
I20260810 06:28:13.602409 30984 leader_election.cc:302] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570; no voters: 11f48fb346c74bf6980b680bb874785c
I20260810 06:28:13.602613 31466 raft_consensus.cc:2781] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:28:13.602838 31466 raft_consensus.cc:686] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 1 LEADER]: Becoming Leader. State: Replica: d55186e037a947418cf0b19a5d455570, State: Running, Role: LEADER
I20260810 06:28:13.603168 31466 consensus_queue.cc:237] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:13.606302 30610 catalog_manager.cc:5495] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 reported cstate change: term changed from 0 to 1, leader changed from <none> to d55186e037a947418cf0b19a5d455570 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "d55186e037a947418cf0b19a5d455570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } health_report { overall_health: HEALTHY } } }
I20260810 06:28:13.635864 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.632248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.4.73.2:44155)
0810 06:28:13.632446 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:13.632450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:13.632466 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:28:13.632549 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:13.632552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:13.632611 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:13.632716 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:28:13.633009 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.633525 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.634427 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:13.634668 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:13.634762 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:13.635054 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:28:13.635505 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:28:13.635609 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:13.635665 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 06:28:13.635716 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 06:28:13.642205 30866 raft_consensus.cc:1261] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Refusing update from remote peer d55186e037a947418cf0b19a5d455570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:28:13.642220 31328 raft_consensus.cc:1261] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 FOLLOWER]: Refusing update from remote peer d55186e037a947418cf0b19a5d455570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:28:13.642908 31479 consensus_queue.cc:1035] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:28:13.643168 31466 consensus_queue.cc:1035] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:28:13.651405 31487 mvcc.cc:205] Tried to move back new op lower bound from 7316862130750910464 to 7316862130601349120. Current Snapshot: MvccSnapshot[applied={T|T < 7316862130750910464}]
I20260810 06:28:13.653602 31485 mvcc.cc:205] Tried to move back new op lower bound from 7316862130750910464 to 7316862130601349120. Current Snapshot: MvccSnapshot[applied={T|T < 7316862130750910464}]
I20260810 06:28:13.659034 30610 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:28:13.659219 30610 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:28:13.661732 30610 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Sending DeleteTablet for 3 replicas of tablet 931fe2191b9c4d118b773a1cfb6aebbb
I20260810 06:28:13.662489 31292 tablet_service.cc:1514] Processing DeleteTablet for tablet 931fe2191b9c4d118b773a1cfb6aebbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:28:13 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:58224
I20260810 06:28:13.662662 31067 tablet_service.cc:1514] Processing DeleteTablet for tablet 931fe2191b9c4d118b773a1cfb6aebbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:28:13 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:45506
I20260810 06:28:13.662662 30829 tablet_service.cc:1514] Processing DeleteTablet for tablet 931fe2191b9c4d118b773a1cfb6aebbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:28:13 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38442
I20260810 06:28:13.663070 31502 tablet_replica.cc:330] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: stopping tablet replica
I20260810 06:28:13.663265 31500 tablet_replica.cc:330] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: stopping tablet replica
I20260810 06:28:13.663355 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 30721
I20260810 06:28:13.663400 31502 raft_consensus.cc:2229] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:28:13.663484 31500 raft_consensus.cc:2229] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:28:13.663755 31502 raft_consensus.cc:2258] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:28:13.663761 31500 raft_consensus.cc:2258] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:28:13.669936 31500 ts_tablet_manager.cc:1905] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:28:13.673998 31500 ts_tablet_manager.cc:1918] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:28:13.674152 31500 log.cc:1196] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/931fe2191b9c4d118b773a1cfb6aebbb
I20260810 06:28:13.674585 31500 ts_tablet_manager.cc:1939] T 931fe2191b9c4d118b773a1cfb6aebbb P d0e5ae51cabe47dea4e583e3c30ec627: Deleting consensus metadata
I20260810 06:28:13.675082 31502 ts_tablet_manager.cc:1905] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:28:13.676082 30587 catalog_manager.cc:4841] TS d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029): tablet 931fe2191b9c4d118b773a1cfb6aebbb (table test-table [id=92947f51ca0646978e784773832337eb]) successfully deleted
I20260810 06:28:13.677624 31502 ts_tablet_manager.cc:1918] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:28:13.677754 31502 log.cc:1196] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/931fe2191b9c4d118b773a1cfb6aebbb
I20260810 06:28:13.678061 31502 ts_tablet_manager.cc:1939] T 931fe2191b9c4d118b773a1cfb6aebbb P d55186e037a947418cf0b19a5d455570: Deleting consensus metadata
I20260810 06:28:13.679159 30585 catalog_manager.cc:4841] TS d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155): tablet 931fe2191b9c4d118b773a1cfb6aebbb (table test-table [id=92947f51ca0646978e784773832337eb]) successfully deleted
W20260810 06:28:13.687320 30583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:40933 (error 108)
I20260810 06:28:13.688082 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 30951
I20260810 06:28:13.688712 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:13.688326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40933 (local address 127.0.0.1:38446)
0810 06:28:13.688497 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:13.688591 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 06:28:13.692850 30583 catalog_manager.cc:4584] TS 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 931fe2191b9c4d118b773a1cfb6aebbb: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40933: connect: Connection refused (error 111)
I20260810 06:28:13.702824 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 31188
I20260810 06:28:13.712877 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 30560
I20260810 06:28:33.731393 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44121
--webserver_interface=127.4.73.62
--webserver_port=38603
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:44121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:28:33.840969 31505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:33.841203 31505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:33.841240 31505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:33.845173 31505 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 06:28:33.845216 31505 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 06:28:33.845230 31505 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 06:28:33.845244 31505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:28:33.845260 31505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:33.845278 31505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:33.845297 31505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:28:33.845317 31505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:28:33.850386 31505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:44121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=38603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:33.851279 31505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:33.852147 31505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:33.865617 31518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:33.865617 31516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:33.865773 31505 server_base.cc:1034] running on GCE node
W20260810 06:28:33.865947 31514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:33.866633 31505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:33.867786 31505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:33.868949 31505 hybrid_clock.cc:648] HybridClock initialized: now 1786343313868929 us; error 38 us; skew 500 ppm
Aug 10 06:28:33 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:33.876654 31505 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:28:33.877626 31505 webserver.cc:460] Webserver started at http://127.4.73.62:38603/ using document root <none> and password file <none>
I20260810 06:28:33.878289 31505 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:33.878407 31505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:33.882395 31505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:28:33.884651 31528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:33.885885 31505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:28:33.886034 31505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "6717415825db4592b97290e57aae85fd"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "16ff874c78a5109b253bce142fa458a3"
server_key_iv: "aca4a9caf644e6671dc2feb3cb228452"
server_key_version: "encryptionkey@0"
I20260810 06:28:33.886650 31505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:33.900184 31505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:33.903563 31505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:33.903841 31505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:33.911046 31505 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:44121
I20260810 06:28:33.911343 31632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:44121 every 8 connection(s)
I20260810 06:28:33.912139 31505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:28:33.916540 31634 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:28:33.918114 31634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Bootstrap starting.
I20260810 06:28:33.919013 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 31505
I20260810 06:28:33.919566 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:40933
--local_ip_for_outbound_sockets=127.4.73.1
--tserver_master_addrs=127.4.73.62:44121
--webserver_port=40215
--webserver_interface=127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:28:33.936048 31634 log.cc:824] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:33.944847 31634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:28:33.945225 31634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Bootstrap complete.
I20260810 06:28:33.949323 31634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:33.949723 31634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6717415825db4592b97290e57aae85fd, State: Initialized, Role: FOLLOWER
I20260810 06:28:33.950253 31634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:33.950402 31634 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:28:33.950498 31634 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:28:33.950616 31634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 1 FOLLOWER]: Advancing to term 2
I20260810 06:28:33.952461 31634 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:33.952783 31634 leader_election.cc:302] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6717415825db4592b97290e57aae85fd; no voters:
I20260810 06:28:33.953114 31634 leader_election.cc:288] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 06:28:33.953194 31648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 06:28:33.953402 31648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 LEADER]: Becoming Leader. State: Replica: 6717415825db4592b97290e57aae85fd, State: Running, Role: LEADER
I20260810 06:28:33.953682 31648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:33.954320 31634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:28:33.954614 31648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6717415825db4592b97290e57aae85fd. Latest consensus state: current_term: 2 leader_uuid: "6717415825db4592b97290e57aae85fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } } }
I20260810 06:28:33.954735 31648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: This master's current role is: LEADER
I20260810 06:28:33.955196 31653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:28:33.955139 31649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6717415825db4592b97290e57aae85fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } } }
I20260810 06:28:33.955406 31649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: This master's current role is: LEADER
I20260810 06:28:33.959904 31653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:28:33.960614 31653 catalog_manager.cc:1259] Loaded cluster ID: bcc839e0182f43c1983e695cc79ae31b
I20260810 06:28:33.960688 31653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:28:33.962378 31653 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:28:33.962978 31653 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Loaded TSK: 0
I20260810 06:28:33.976864 31653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Generated new TSK 1
I20260810 06:28:33.977111 31653 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:28:34.055714 31638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:34.056087 31638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:34.056170 31638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:34.060527 31638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:34.060582 31638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:34.060665 31638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:28:34.065801 31638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:40933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=40215
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:34.066813 31638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:34.067797 31638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:34.077369 31687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:34.077376 31682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:34.077369 31684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:34.078500 31638 server_base.cc:1034] running on GCE node
I20260810 06:28:34.078919 31638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:34.079526 31638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:34.080714 31638 hybrid_clock.cc:648] HybridClock initialized: now 1786343314080695 us; error 47 us; skew 500 ppm
Aug 10 06:28:34 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:34.083523 31638 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:28:34.084556 31638 webserver.cc:460] Webserver started at http://127.4.73.1:40215/ using document root <none> and password file <none>
I20260810 06:28:34.085099 31638 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:34.085170 31638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:34.089036 31638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:28:34.091408 31698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:34.092859 31638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:28:34.092998 31638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "11f48fb346c74bf6980b680bb874785c"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "f76fe948902111d61332901fb6c909ee"
server_key_iv: "6876258f40f79573bda1734f8145196c"
server_key_version: "encryptionkey@0"
I20260810 06:28:34.093672 31638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:34.110971 31638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:34.114147 31638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:34.114351 31638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:34.115010 31638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:34.116118 31709 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:28:34.120399 31638 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:28:34.120478 31638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 06:28:34.120553 31638 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:28:34.122169 31638 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:28:34.122225 31638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 06:28:34.122485 31709 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Bootstrap starting.
I20260810 06:28:34.133270 31638 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:40933
I20260810 06:28:34.133404 31914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:40933 every 8 connection(s)
I20260810 06:28:34.134526 31638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:28:34.137897 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 31638
I20260810 06:28:34.138396 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:44155
--local_ip_for_outbound_sockets=127.4.73.2
--tserver_master_addrs=127.4.73.62:44121
--webserver_port=37083
--webserver_interface=127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:28:34.143784 31709 log.cc:824] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:28:34 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:28:34.152196 31709 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:28:34.152722 31709 tablet_bootstrap.cc:492] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Bootstrap complete.
I20260810 06:28:34.154170 31709 ts_tablet_manager.cc:1397] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Time spent bootstrapping tablet: real 0.032s user 0.026s sys 0.005s
I20260810 06:28:34.154605 31923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.136593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:35553)
0810 06:28:34.137183 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.137223 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.138239 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.139870 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.139880 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.140437 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:34.141078 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.141099 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.143127 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.143130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.144065 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.144080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.145212 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.145216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.145401 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.146111 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:34.146131 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:34.148176 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:34.150753 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.150761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.150777 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.151175 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.151597 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.151603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.151605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.151798 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.152583 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:34.152606 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:34.152936 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.153149 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.153546 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":178,"threads_started":1}
I20260810 06:28:34.154623 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.136600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:35553 (local address 127.4.73.62:44121)
0810 06:28:34.137013 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.137034 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.137349 (+ 315us) server_negotiation.cc:407] Connection header received
0810 06:28:34.138983 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.138998 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.139390 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.139752 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:34.141293 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.142962 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.144215 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.145064 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.145245 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.148442 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:34.148448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:34.148462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:34.148495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:34.150620 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.151324 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.151329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.151335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.151485 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.151937 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.151941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.151942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.152422 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:34.152590 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.153327 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.153630 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260810 06:28:34.157701 31916 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44121
I20260810 06:28:34.157923 31916 heartbeater.cc:464] Registering TS with master...
I20260810 06:28:34.157545 31709 raft_consensus.cc:348] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:34.158054 31709 raft_consensus.cc:729] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f48fb346c74bf6980b680bb874785c, State: Initialized, Role: FOLLOWER
I20260810 06:28:34.158423 31916 heartbeater.cc:511] Master 127.4.73.62:44121 requested a full tablet report, sending...
I20260810 06:28:34.158545 31709 consensus_queue.cc:260] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } }
I20260810 06:28:34.159543 31709 ts_tablet_manager.cc:1428] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:28:34.161051 31562 ts_manager.cc:194] Registered new tserver with Master: 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933)
I20260810 06:28:34.164283 31562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:35553
I20260810 06:28:34.166656 31916 heartbeater.cc:503] Master 127.4.73.62:44121 was elected leader, sending a full tablet report...
Aug 10 06:28:34 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343313, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.1@KRBTEST.COM
I20260810 06:28:34.174073 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.163656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40933 (local address 127.0.0.1:60324)
0810 06:28:34.164376 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.164433 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.164617 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.165217 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.165222 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.165618 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:34.165749 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.165757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.167003 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.167007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.167908 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.167920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.168555 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.168571 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.168688 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.169237 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:34.169252 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:34.170697 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:34.172554 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.172562 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.172566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.172767 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.173049 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.173051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.173052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.173122 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.173619 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:34.173623 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:34.173715 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.173822 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.173957 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":109,"threads_started":1}
I20260810 06:28:34.174228 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.163736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.4.73.1:40933)
0810 06:28:34.164084 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.164095 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.164524 (+ 429us) server_negotiation.cc:407] Connection header received
0810 06:28:34.164769 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.164774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.164948 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.165096 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:34.165891 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.166861 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.168013 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.168458 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.168540 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.170825 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:34.170834 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:34.170845 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:34.170867 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:34.172458 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.172871 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.172876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.172878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.172962 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.173205 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.173208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.173209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.173508 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:34.173636 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.173925 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.174078 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260810 06:28:34.175036 31800 tablet_service.cc:1514] Processing DeleteTablet for tablet 931fe2191b9c4d118b773a1cfb6aebbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:28:13 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260810 06:28:34.175776 31931 tablet_replica.cc:330] stopping tablet replica
I20260810 06:28:34.176055 31931 raft_consensus.cc:2229] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:28:34.176268 31931 raft_consensus.cc:2258] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:28:34.177172 31931 ts_tablet_manager.cc:1905] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:28:34.180676 31931 ts_tablet_manager.cc:1918] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:28:34.180785 31931 log.cc:1196] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal/wals/931fe2191b9c4d118b773a1cfb6aebbb
I20260810 06:28:34.181069 31931 ts_tablet_manager.cc:1939] T 931fe2191b9c4d118b773a1cfb6aebbb P 11f48fb346c74bf6980b680bb874785c: Deleting consensus metadata
I20260810 06:28:34.182618 31530 catalog_manager.cc:4841] TS 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933): tablet 931fe2191b9c4d118b773a1cfb6aebbb (table test-table [id=92947f51ca0646978e784773832337eb]) successfully deleted
W20260810 06:28:34.259688 31924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:34.259941 31924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:34.260023 31924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:34.265069 31924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:34.265151 31924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:34.265266 31924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:28:34.272677 31924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:44155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=37083
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:34.273968 31924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:34.275050 31924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:34.284850 31941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:34.284896 31943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:34.284847 31939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:34.285151 31924 server_base.cc:1034] running on GCE node
I20260810 06:28:34.285605 31924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:34.286166 31924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:34.287331 31924 hybrid_clock.cc:648] HybridClock initialized: now 1786343314287319 us; error 23 us; skew 500 ppm
Aug 10 06:28:34 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:34.290171 31924 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:28:34.291126 31924 webserver.cc:460] Webserver started at http://127.4.73.2:37083/ using document root <none> and password file <none>
I20260810 06:28:34.291657 31924 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:34.291756 31924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:34.295461 31924 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:28:34.297645 31953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:34.298883 31924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:28:34.299014 31924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "d55186e037a947418cf0b19a5d455570"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "92a5151ef7f809c5462c17c973c47981"
server_key_iv: "5052e9d1def7db36453ec568d57c428e"
server_key_version: "encryptionkey@0"
I20260810 06:28:34.299517 31924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:34.311969 31924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:34.314662 31924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:34.314844 31924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:34.315354 31924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:34.316069 31924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:28:34.316112 31924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:34.316174 31924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:28:34.316210 31924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:34.325624 31924 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:44155
I20260810 06:28:34.325644 32130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:44155 every 8 connection(s)
I20260810 06:28:34.326586 31924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:28:34.328814 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 31924
I20260810 06:28:34.329303 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:38029
--local_ip_for_outbound_sockets=127.4.73.3
--tserver_master_addrs=127.4.73.62:44121
--webserver_port=46403
--webserver_interface=127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 06:28:34 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:28:34.341920 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.328467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:54439 (local address 127.4.73.62:44121)
0810 06:28:34.328656 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.328661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.329873 (+ 1212us) server_negotiation.cc:407] Connection header received
0810 06:28:34.331198 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.331203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.331264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.331366 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:34.332545 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.333407 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.334649 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.335086 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.335217 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.338097 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:34.338100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:34.338102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:34.338129 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:34.339572 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.340216 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.340219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.340220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.340298 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.340755 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.340757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.340758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.340934 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:34.341006 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.341675 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.341796 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:28:34.342952 32139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.328886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:54439)
0810 06:28:34.329683 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.329739 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.330952 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.331516 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.331522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.331846 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:34.332395 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.332411 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.333538 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.333541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.334518 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.334528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.335237 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.335240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.335379 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.336016 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:34.336033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:34.337922 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:34.339692 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.339698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.339712 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.340107 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.340448 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.340455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.340458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.340650 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.341043 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:34.341053 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:34.341396 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.341618 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.341965 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":106,"threads_started":1}
I20260810 06:28:34.344446 32136 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44121
I20260810 06:28:34.344697 32136 heartbeater.cc:464] Registering TS with master...
I20260810 06:28:34.345171 32136 heartbeater.cc:511] Master 127.4.73.62:44121 requested a full tablet report, sending...
I20260810 06:28:34.346259 31562 ts_manager.cc:194] Registered new tserver with Master: d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
I20260810 06:28:34.346808 31562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:54439
W20260810 06:28:34.443063 32140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:34.443351 32140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:34.443434 32140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:34.447448 32140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:34.447520 32140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:34.447594 32140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:28:34.452625 32140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:38029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=46403
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:34.453563 32140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:34.454550 32140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:34.463887 32151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:34.463971 32156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:34.464036 32140 server_base.cc:1034] running on GCE node
W20260810 06:28:34.463986 32148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:34.464628 32140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:34.465185 32140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:34.466354 32140 hybrid_clock.cc:648] HybridClock initialized: now 1786343314466322 us; error 29 us; skew 500 ppm
Aug 10 06:28:34 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:34.469259 32140 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:28:34.470052 32140 webserver.cc:460] Webserver started at http://127.4.73.3:46403/ using document root <none> and password file <none>
I20260810 06:28:34.470573 32140 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:34.470642 32140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:34.474283 32140 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260810 06:28:34.476416 32165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:34.477579 32140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:28:34.477712 32140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "d0e5ae51cabe47dea4e583e3c30ec627"
format_stamp: "Formatted at 2026-08-10 06:28:13 on dist-test-slave-45dx"
server_key: "14f6bf36a40f51e10393c442bf490146"
server_key_iv: "33369568bbe61fcbde87c6e99a7ae7f6"
server_key_version: "encryptionkey@0"
I20260810 06:28:34.478286 32140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:34.492789 32140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:34.495921 32140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:34.496107 32140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:34.496773 32140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:34.497550 32140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:28:34.497615 32140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:34.497695 32140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:28:34.497745 32140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:34.506297 32140 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:38029
I20260810 06:28:34.506377 32371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:38029 every 8 connection(s)
I20260810 06:28:34.507319 32140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
Aug 10 06:28:34 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343314, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:28:34.516960 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 32140
I20260810 06:28:34.520102 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.509070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:59185 (local address 127.4.73.62:44121)
0810 06:28:34.509212 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.509215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.510188 (+ 973us) server_negotiation.cc:407] Connection header received
0810 06:28:34.510973 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.510976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.511023 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.511098 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:34.512184 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.512777 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.513854 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.514067 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.514142 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.516440 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:34.516443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:34.516445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:34.516470 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:34.518081 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.518568 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.518573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.518575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.518676 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.519051 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.519054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.519056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.519247 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:34.519374 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.519818 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.519934 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:28:34.520884 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.509401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:59185)
0810 06:28:34.510053 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.510089 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.510805 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.511254 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.511260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.511590 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:34.512031 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.512043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.512884 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.512886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.513750 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.513759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.514187 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.514189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.514317 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.514844 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:34.514859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:34.516228 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:34.518207 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.518218 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.518228 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.518462 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.518783 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:34.518790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:34.518792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:34.518926 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:34.519374 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:34.519381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:34.519592 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.519772 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.520071 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":110,"threads_started":1}
I20260810 06:28:34.522042 32372 heartbeater.cc:347] Connected to a master server at 127.4.73.62:44121
I20260810 06:28:34.522342 32372 heartbeater.cc:464] Registering TS with master...
I20260810 06:28:34.522789 32372 heartbeater.cc:511] Master 127.4.73.62:44121 requested a full tablet report, sending...
I20260810 06:28:34.523838 31562 ts_manager.cc:194] Registered new tserver with Master: d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029)
I20260810 06:28:34.524571 31562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:59185
I20260810 06:28:34.525970 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.517547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.4.73.62:44121)
0810 06:28:34.517766 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.517770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.517974 (+ 204us) server_negotiation.cc:407] Connection header received
0810 06:28:34.518133 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.518136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.518186 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.518277 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:34.519040 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.519560 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.520655 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.520882 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.520988 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.522538 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:34.522542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:34.522544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:34.522573 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:34.524267 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.524752 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.524755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.524757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.524846 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.525156 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.525159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.525161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.525331 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:34.525557 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.525697 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.525810 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260810 06:28:34.527451 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:28:34.531013 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.528050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.4.73.62:44121)
0810 06:28:34.528158 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.528162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.528341 (+ 179us) server_negotiation.cc:407] Connection header received
0810 06:28:34.528529 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.528531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.528611 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.528687 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:28:34.528884 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.529326 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.530062 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.530259 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.530330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.530615 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:28:34.530722 (+ 107us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 06:28:34.530900 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36042: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:28:34.531080 32379 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36042: token expired
W20260810 06:28:34.531432 31432 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.73.62:44121)
I20260810 06:28:34.537961 32379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.531781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.4.73.62:44121)
0810 06:28:34.531889 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.531892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.531903 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:28:34.531979 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.531981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.532009 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.532087 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:34.532685 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.533111 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.533840 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.534029 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.534101 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.535201 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:34.535203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:34.535205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:34.535225 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:34.536733 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.537074 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.537076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.537077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.537147 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:34.537359 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:34.537361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:34.537362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:34.537509 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:34.537598 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.537742 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.537818 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:28:34.540007 31562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:28:34.543047 31562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:28:34.550331 31800 tablet_service.cc:1467] Processing CreateTablet for tablet 1e005e56a9714dac9c8687c81122b4ad (DEFAULT_TABLE table=test-table [id=6733ffd5570146728fda96faa702c793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:28:34.550992 31800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e005e56a9714dac9c8687c81122b4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:28:34.553925 31709 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Bootstrap starting.
I20260810 06:28:34.554909 31709 tablet_bootstrap.cc:654] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Neither blocks nor log segments found. Creating new log.
I20260810 06:28:34.554901 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.549284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:38029 (local address 127.0.0.1:42964)
0810 06:28:34.549397 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.549411 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.549485 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.550412 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.550415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.550433 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:34.550521 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.550527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.551767 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.551770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.552895 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.552904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.553700 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.553704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.554524 (+ 820us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.554568 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.554648 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.554726 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":681}
I20260810 06:28:34.554951 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.549581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42964 (local address 127.4.73.3:38029)
0810 06:28:34.549899 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.549911 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.549933 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:28:34.550008 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.550013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.550214 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.550428 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:34.550600 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.551638 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.553015 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.553574 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.553701 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.554227 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.554687 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.554794 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":137,"threads_started":1}
I20260810 06:28:34.556257 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.549881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:44155 (local address 127.0.0.1:51702)
0810 06:28:34.550159 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.550171 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.550253 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.550929 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.550932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.550949 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:34.551133 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.551142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.552691 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.552694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.553888 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.553901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.555769 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.555771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.555878 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.555900 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.555990 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.556070 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":29,"thread_start_us":94,"threads_started":1}
I20260810 06:28:34.556731 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.550088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51702 (local address 127.4.73.2:44155)
0810 06:28:34.550484 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.550496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.550518 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:28:34.550606 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.550610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.550790 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.550932 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:34.551346 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.552563 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.554105 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.555529 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.555595 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.555691 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.556535 (+ 844us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.556623 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":82,"threads_started":1}
I20260810 06:28:34.558169 32248 tablet_service.cc:1467] Processing CreateTablet for tablet 1e005e56a9714dac9c8687c81122b4ad (DEFAULT_TABLE table=test-table [id=6733ffd5570146728fda96faa702c793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:28:34.559275 32248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e005e56a9714dac9c8687c81122b4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:28:34.559368 32023 tablet_service.cc:1467] Processing CreateTablet for tablet 1e005e56a9714dac9c8687c81122b4ad (DEFAULT_TABLE table=test-table [id=6733ffd5570146728fda96faa702c793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:28:34.560388 32023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e005e56a9714dac9c8687c81122b4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:28:34.560832 31709 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: No bootstrap required, opened a new log
I20260810 06:28:34.560976 31709 ts_tablet_manager.cc:1397] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 06:28:34.561510 31709 raft_consensus.cc:348] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.561650 31709 raft_consensus.cc:374] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:28:34.561700 31709 raft_consensus.cc:729] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f48fb346c74bf6980b680bb874785c, State: Initialized, Role: FOLLOWER
I20260810 06:28:34.561862 31709 consensus_queue.cc:260] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.562206 31709 ts_tablet_manager.cc:1428] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 06:28:34.567659 32397 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Bootstrap starting.
I20260810 06:28:34.568182 32399 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Bootstrap starting.
I20260810 06:28:34.570039 32397 tablet_bootstrap.cc:654] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Neither blocks nor log segments found. Creating new log.
I20260810 06:28:34.570672 32399 tablet_bootstrap.cc:654] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Neither blocks nor log segments found. Creating new log.
I20260810 06:28:34.571434 32399 log.cc:824] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:34.571434 32397 log.cc:824] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:34.573529 32397 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: No bootstrap required, opened a new log
I20260810 06:28:34.573719 32399 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: No bootstrap required, opened a new log
I20260810 06:28:34.573727 32397 ts_tablet_manager.cc:1397] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 06:28:34.573908 32399 ts_tablet_manager.cc:1397] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:28:34.577685 32397 raft_consensus.cc:348] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.577685 32399 raft_consensus.cc:348] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.577967 32399 raft_consensus.cc:374] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:28:34.577966 32397 raft_consensus.cc:374] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:28:34.578030 32397 raft_consensus.cc:729] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e5ae51cabe47dea4e583e3c30ec627, State: Initialized, Role: FOLLOWER
I20260810 06:28:34.578030 32399 raft_consensus.cc:729] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55186e037a947418cf0b19a5d455570, State: Initialized, Role: FOLLOWER
I20260810 06:28:34.578547 32397 consensus_queue.cc:260] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.578547 32399 consensus_queue.cc:260] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.579231 32136 heartbeater.cc:503] Master 127.4.73.62:44121 was elected leader, sending a full tablet report...
I20260810 06:28:34.579555 32399 ts_tablet_manager.cc:1428] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:28:34.579555 32397 ts_tablet_manager.cc:1428] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:28:34.580186 32372 heartbeater.cc:503] Master 127.4.73.62:44121 was elected leader, sending a full tablet report...
W20260810 06:28:34.635749 31918 tablet.cc:2369] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:28:34.690662 32403 raft_consensus.cc:482] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:28:34.690898 32403 raft_consensus.cc:504] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.692063 32403 leader_election.cc:288] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155), 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933)
I20260810 06:28:34.696877 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.692454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:46419 (local address 127.4.73.2:44155)
0810 06:28:34.692580 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.692587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.692608 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:28:34.692692 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.692695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.692748 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.692841 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:34.693115 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.693927 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.695480 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.696117 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.696290 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.696396 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.696674 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.696751 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:28:34.696895 32377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.692298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:44155 (local address 127.4.73.3:46419)
0810 06:28:34.692470 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.692490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.692584 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.692859 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.692864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.692884 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:34.693003 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.693008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.694043 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.694045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.695353 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.695366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.696284 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.696288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.696439 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.696564 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.696645 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.696761 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":132}
I20260810 06:28:34.696926 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.692521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:49923 (local address 127.4.73.1:40933)
0810 06:28:34.692771 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.692778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.692794 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:28:34.692951 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.692955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.693016 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.693140 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:28:34.693374 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.694100 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.695480 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.696194 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.696304 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.696511 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.696674 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.696788 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260810 06:28:34.697409 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.692337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40933 (local address 127.4.73.3:49923)
0810 06:28:34.692719 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.692729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:34.692819 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:34.693161 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:34.693163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:34.693179 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:28:34.693261 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.693265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.694191 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.694193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.695366 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:34.695373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.696353 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.696357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:34.696471 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.696563 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 06:28:34.696645 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 06:28:34.697253 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":543,"thread_start_us":144,"threads_started":1}
I20260810 06:28:34.697901 32060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e005e56a9714dac9c8687c81122b4ad" candidate_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55186e037a947418cf0b19a5d455570" is_pre_election: true
I20260810 06:28:34.697973 31831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e005e56a9714dac9c8687c81122b4ad" candidate_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f48fb346c74bf6980b680bb874785c" is_pre_election: true
I20260810 06:28:34.698213 32060 raft_consensus.cc:2454] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e5ae51cabe47dea4e583e3c30ec627 in term 0.
I20260810 06:28:34.698257 31831 raft_consensus.cc:2454] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e5ae51cabe47dea4e583e3c30ec627 in term 0.
I20260810 06:28:34.698734 32166 leader_election.cc:302] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c, d0e5ae51cabe47dea4e583e3c30ec627; no voters:
I20260810 06:28:34.699007 32403 raft_consensus.cc:2781] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:28:34.699105 32403 raft_consensus.cc:482] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:28:34.699162 32403 raft_consensus.cc:3037] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:28:34.700217 32403 raft_consensus.cc:504] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.700584 32403 leader_election.cc:288] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [CANDIDATE]: Term 1 election: Requested vote from peers d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155), 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933)
I20260810 06:28:34.700956 32060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e005e56a9714dac9c8687c81122b4ad" candidate_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d55186e037a947418cf0b19a5d455570"
I20260810 06:28:34.701085 32060 raft_consensus.cc:3037] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:28:34.702152 32060 raft_consensus.cc:2454] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e5ae51cabe47dea4e583e3c30ec627 in term 1.
I20260810 06:28:34.702454 31831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e005e56a9714dac9c8687c81122b4ad" candidate_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11f48fb346c74bf6980b680bb874785c"
I20260810 06:28:34.702510 32167 leader_election.cc:302] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570; no voters:
I20260810 06:28:34.702595 31831 raft_consensus.cc:3037] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:28:34.703570 31831 raft_consensus.cc:2454] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e5ae51cabe47dea4e583e3c30ec627 in term 1.
I20260810 06:28:34.703805 32403 raft_consensus.cc:2781] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:28:34.704056 32403 raft_consensus.cc:686] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 LEADER]: Becoming Leader. State: Replica: d0e5ae51cabe47dea4e583e3c30ec627, State: Running, Role: LEADER
I20260810 06:28:34.704448 32403 consensus_queue.cc:237] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:34.707561 31562 catalog_manager.cc:5495] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } health_report { overall_health: UNKNOWN } } }
I20260810 06:28:34.742436 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.738928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.4.73.3:38029)
0810 06:28:34.739077 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:34.739081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:34.739097 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:28:34.739199 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:34.739201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:34.739256 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:34.739377 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:28:34.739623 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.740110 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.740970 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:34.741188 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:34.741300 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:34.741610 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:28:34.742011 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:28:34.742143 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:34.742204 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 06:28:34.742260 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:28:34.748258 31831 raft_consensus.cc:1261] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Refusing update from remote peer d0e5ae51cabe47dea4e583e3c30ec627: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:28:34.748262 32060 raft_consensus.cc:1261] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Refusing update from remote peer d0e5ae51cabe47dea4e583e3c30ec627: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:28:34.749055 32408 consensus_queue.cc:1035] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:28:34.749265 32403 consensus_queue.cc:1035] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:28:34.763634 31560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:28:34.763836 31560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:28:34.766286 31560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Sending DeleteTablet for 3 replicas of tablet 1e005e56a9714dac9c8687c81122b4ad
I20260810 06:28:34.767007 32248 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e005e56a9714dac9c8687c81122b4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:28:34 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:42964
I20260810 06:28:34.767045 32023 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e005e56a9714dac9c8687c81122b4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:28:34 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:51702
I20260810 06:28:34.767485 31800 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e005e56a9714dac9c8687c81122b4ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:28:34 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60324
I20260810 06:28:34.767774 32437 tablet_replica.cc:330] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: stopping tablet replica
I20260810 06:28:34.767530 32435 tablet_replica.cc:330] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: stopping tablet replica
I20260810 06:28:34.767869 32438 tablet_replica.cc:330] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: stopping tablet replica
I20260810 06:28:34.767917 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 31638
I20260810 06:28:34.768007 32438 raft_consensus.cc:2229] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:28:34.768141 32438 raft_consensus.cc:2258] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:28:34.768477 32437 raft_consensus.cc:2229] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:28:34.768797 32435 raft_consensus.cc:2229] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:28:34.769217 32435 raft_consensus.cc:2258] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:28:34.769657 32437 raft_consensus.cc:2258] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:28:34.774435 32435 ts_tablet_manager.cc:1905] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:28:34.776933 32437 ts_tablet_manager.cc:1905] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:28:34.780339 32437 ts_tablet_manager.cc:1918] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:28:34.782764 32435 ts_tablet_manager.cc:1918] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:28:34.782889 32435 log.cc:1196] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/1e005e56a9714dac9c8687c81122b4ad
I20260810 06:28:34.783236 32435 ts_tablet_manager.cc:1939] T 1e005e56a9714dac9c8687c81122b4ad P d55186e037a947418cf0b19a5d455570: Deleting consensus metadata
I20260810 06:28:34.784447 32437 log.cc:1196] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/1e005e56a9714dac9c8687c81122b4ad
I20260810 06:28:34.784596 31532 catalog_manager.cc:4841] TS d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155): tablet 1e005e56a9714dac9c8687c81122b4ad (table test-table [id=6733ffd5570146728fda96faa702c793]) successfully deleted
I20260810 06:28:34.784834 32437 ts_tablet_manager.cc:1939] T 1e005e56a9714dac9c8687c81122b4ad P d0e5ae51cabe47dea4e583e3c30ec627: Deleting consensus metadata
I20260810 06:28:34.785785 31535 catalog_manager.cc:4841] TS d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029): tablet 1e005e56a9714dac9c8687c81122b4ad (table test-table [id=6733ffd5570146728fda96faa702c793]) successfully deleted
W20260810 06:28:34.787539 31530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:40933 (error 108)
I20260810 06:28:34.787804 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 31924
I20260810 06:28:34.796416 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:34.796035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40933 (local address 127.0.0.1:60330)
0810 06:28:34.796180 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:34.796275 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 06:28:34.796680 31530 catalog_manager.cc:4584] TS 11f48fb346c74bf6980b680bb874785c (127.4.73.1:40933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e005e56a9714dac9c8687c81122b4ad: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40933: connect: Connection refused (error 111)
I20260810 06:28:34.804481 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 32140
I20260810 06:28:34.816437 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 31505
I20260810 06:28:54.828033 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44121
--webserver_interface=127.4.73.62
--webserver_port=38603
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:44121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:28:54.941938 32445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:54.942188 32445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:54.942226 32445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:54.946223 32445 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 06:28:54.946271 32445 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 06:28:54.946287 32445 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 06:28:54.946300 32445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:28:54.946357 32445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:54.946401 32445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:54.946437 32445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:28:54.946476 32445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:28:54.951524 32445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:44121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=38603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:54.952546 32445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:54.953572 32445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:54.962425 32462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:54.962463 32445 server_base.cc:1034] running on GCE node
W20260810 06:28:54.962486 32458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:54.962729 32456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:54.963356 32445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:54.964485 32445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:54.965653 32445 hybrid_clock.cc:648] HybridClock initialized: now 1786343334965636 us; error 31 us; skew 500 ppm
Aug 10 06:28:54 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343334, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:54.972915 32445 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:28:54.973831 32445 webserver.cc:460] Webserver started at http://127.4.73.62:38603/ using document root <none> and password file <none>
I20260810 06:28:54.974375 32445 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:54.974444 32445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:54.978394 32445 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:28:54.980541 32476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:54.981783 32445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:28:54.981904 32445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "6717415825db4592b97290e57aae85fd"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "16ff874c78a5109b253bce142fa458a3"
server_key_iv: "aca4a9caf644e6671dc2feb3cb228452"
server_key_version: "encryptionkey@0"
I20260810 06:28:54.982412 32445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:54.995599 32445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:54.998260 32445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:54.998405 32445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:55.005504 32445 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:44121
I20260810 06:28:55.005813 32559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:44121 every 8 connection(s)
I20260810 06:28:55.006584 32445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:28:55.010810 32562 sys_catalog.cc:263] Verifying existing consensus state
I20260810 06:28:55.012473 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Bootstrap starting.
I20260810 06:28:55.016343 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 32445
I20260810 06:28:55.016801 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:40933
--local_ip_for_outbound_sockets=127.4.73.1
--tserver_master_addrs=127.4.73.62:44121
--webserver_port=40215
--webserver_interface=127.4.73.1
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:28:55.030323 32562 log.cc:824] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:55.041520 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:28:55.041922 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Bootstrap complete.
I20260810 06:28:55.046121 32562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:55.046532 32562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6717415825db4592b97290e57aae85fd, State: Initialized, Role: FOLLOWER
I20260810 06:28:55.047060 32562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:55.047204 32562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:28:55.047290 32562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:28:55.047412 32562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 2 FOLLOWER]: Advancing to term 3
I20260810 06:28:55.049273 32562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:55.049602 32562 leader_election.cc:302] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6717415825db4592b97290e57aae85fd; no voters:
I20260810 06:28:55.049916 32562 leader_election.cc:288] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 06:28:55.050045 32570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 3 FOLLOWER]: Leader election won for term 3
I20260810 06:28:55.050288 32570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [term 3 LEADER]: Becoming Leader. State: Replica: 6717415825db4592b97290e57aae85fd, State: Running, Role: LEADER
I20260810 06:28:55.050647 32570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } }
I20260810 06:28:55.051095 32562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:28:55.052029 32572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6717415825db4592b97290e57aae85fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } } }
I20260810 06:28:55.052168 32572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: This master's current role is: LEADER
I20260810 06:28:55.052436 32573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6717415825db4592b97290e57aae85fd. Latest consensus state: current_term: 3 leader_uuid: "6717415825db4592b97290e57aae85fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6717415825db4592b97290e57aae85fd" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 44121 } } }
I20260810 06:28:55.052515 32573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd [sys.catalog]: This master's current role is: LEADER
I20260810 06:28:55.052979 32583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:28:55.058193 32583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:28:55.058910 32583 catalog_manager.cc:1259] Loaded cluster ID: bcc839e0182f43c1983e695cc79ae31b
I20260810 06:28:55.058974 32583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:28:55.061523 32583 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:28:55.062309 32583 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Loaded TSK: 0
I20260810 06:28:55.062398 32583 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6717415825db4592b97290e57aae85fd: Loaded TSK: 1
F20260810 06:28:55.063174 32583 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1ccecdb9ad google::LogMessage::Fail() at ??:0
@ 0x7f1ccecde88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1ccecdb4de google::LogMessage::Flush() at ??:0
@ 0x7f1ccecdca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1cd199f18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1cd1997f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1cd96447bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1cd961e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1cd961f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f1cd9660df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1cd967f82b std::function<>::operator()() at ??:0
@ 0x7f1cd961eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1cd961ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1cd961c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1cd9660640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1cd0f2d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1cd0f2dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1cd0f2f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1cd0f1e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1cd25626db start_thread at ??:0
@ 0x7f1ccd7d961f clone at ??:0
W20260810 06:28:55.169934 32568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:55.170176 32568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:55.170248 32568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:55.174316 32568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:55.174373 32568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:55.174458 32568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:28:55.179747 32568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:40933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=40215
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:55.180878 32568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:55.181947 32568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:55.191919 32642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:55.191906 32641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:55.191890 32645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:55.192416 32568 server_base.cc:1034] running on GCE node
I20260810 06:28:55.192807 32568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:55.193429 32568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:55.194620 32568 hybrid_clock.cc:648] HybridClock initialized: now 1786343335194581 us; error 58 us; skew 500 ppm
Aug 10 06:28:55 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:55.197638 32568 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:28:55.198529 32568 webserver.cc:460] Webserver started at http://127.4.73.1:40215/ using document root <none> and password file <none>
I20260810 06:28:55.199093 32568 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:55.199185 32568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:55.203048 32568 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 06:28:55.205408 32655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:55.206672 32568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:28:55.206806 32568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "11f48fb346c74bf6980b680bb874785c"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "f76fe948902111d61332901fb6c909ee"
server_key_iv: "6876258f40f79573bda1734f8145196c"
server_key_version: "encryptionkey@0"
I20260810 06:28:55.207340 32568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:55.220481 32568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:55.223529 32568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:55.223721 32568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:55.224375 32568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:55.225531 32664 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 06:28:55.229606 32568 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 06:28:55.229672 32568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260810 06:28:55.229717 32568 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 06:28:55.231383 32568 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 06:28:55.231441 32568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 06:28:55.231709 32664 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Bootstrap starting.
I20260810 06:28:55.242522 32568 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:40933
I20260810 06:28:55.242743 386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:40933 every 8 connection(s)
I20260810 06:28:55.243714 32568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:28:55.244728 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 32568
I20260810 06:28:55.245187 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:44155
--local_ip_for_outbound_sockets=127.4.73.2
--tserver_master_addrs=127.4.73.62:44121
--webserver_port=37083
--webserver_interface=127.4.73.2
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:28:55.256852 32664 log.cc:824] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Log is configured to *not* fsync() on all Append() calls
I20260810 06:28:55.265877 32664 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 06:28:55.266330 32664 tablet_bootstrap.cc:492] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Bootstrap complete.
I20260810 06:28:55.267791 32664 ts_tablet_manager.cc:1397] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Time spent bootstrapping tablet: real 0.036s user 0.019s sys 0.010s
I20260810 06:28:55.270834 32664 raft_consensus.cc:348] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:55.271351 32664 raft_consensus.cc:729] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f48fb346c74bf6980b680bb874785c, State: Initialized, Role: FOLLOWER
I20260810 06:28:55.271847 32664 consensus_queue.cc:260] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:55.272899 32664 ts_tablet_manager.cc:1428] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260810 06:28:55.298039 391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:28:55.246108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:47061)
0810 06:28:55.246701 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.246763 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:55.247943 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:55.297267 (+ 49324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":168,"threads_started":1}
W20260810 06:28:55.298566 387 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:44121 (0 consecutive failures): Network error: Failed to ping master at 127.4.73.62:44121: Client connection negotiation failed: client connection to 127.4.73.62:44121: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 06:28:55.299341 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.299104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:51727)
0810 06:28:55.299225 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.299287 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 06:28:55.300007 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.299822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:60507)
0810 06:28:55.299918 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.299962 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 06:28:55.300181 387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:28:55.365273 393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:55.365521 393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:55.365562 393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:55.369335 393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:55.369385 393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:55.369436 393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:28:55.374240 393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:44155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=37083
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:55.375134 393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:55.376046 393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:55.385453 413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:55.385480 405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:55.385604 393 server_base.cc:1034] running on GCE node
W20260810 06:28:55.385480 408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:55.386116 393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:55.386755 393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:55.387946 393 hybrid_clock.cc:648] HybridClock initialized: now 1786343335387936 us; error 28 us; skew 500 ppm
Aug 10 06:28:55 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:55.390626 393 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:28:55.391472 393 webserver.cc:460] Webserver started at http://127.4.73.2:37083/ using document root <none> and password file <none>
I20260810 06:28:55.391994 393 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:55.392083 393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:55.395587 393 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:28:55.397550 425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:55.398734 393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:28:55.398882 393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "d55186e037a947418cf0b19a5d455570"
format_stamp: "Formatted at 2026-08-10 06:28:12 on dist-test-slave-45dx"
server_key: "92a5151ef7f809c5462c17c973c47981"
server_key_iv: "5052e9d1def7db36453ec568d57c428e"
server_key_version: "encryptionkey@0"
I20260810 06:28:55.399433 393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:55.424772 393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:55.427980 393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:55.428202 393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:55.428877 393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:55.429726 393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:28:55.429800 393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:55.429877 393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:28:55.430078 393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:55.439487 393 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:44155
I20260810 06:28:55.439517 621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:44155 every 8 connection(s)
I20260810 06:28:55.440527 393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:28:55.443444 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 393
I20260810 06:28:55.443957 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:38029
--local_ip_for_outbound_sockets=127.4.73.3
--tserver_master_addrs=127.4.73.62:44121
--webserver_port=46403
--webserver_interface=127.4.73.3
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:33273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:28:55.445030 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.442609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:49587)
0810 06:28:55.443066 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.443217 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":104,"threads_started":1}
W20260810 06:28:55.446411 622 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:44121 (0 consecutive failures): Network error: Failed to ping master at 127.4.73.62:44121: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
I20260810 06:28:55.447073 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.446828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:44679)
0810 06:28:55.446950 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.447019 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 06:28:55.447711 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.447504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:40527)
0810 06:28:55.447605 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.447651 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 06:28:55.447952 622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 06:28:55.555516 628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:28:55.555753 628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:28:55.555795 628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:28:55.559739 628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:28:55.559796 628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:28:55.559850 628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:28:55.565083 628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:33273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:38029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=46403
--tserver_master_addrs=127.4.73.62:44121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:28:55.566073 628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:28:55.567049 628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:28:55.576272 635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:28:55.576632 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:55.576587 628 server_base.cc:1034] running on GCE node
W20260810 06:28:55.576771 636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:28:55.577126 628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:28:55.577703 628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:28:55.578868 628 hybrid_clock.cc:648] HybridClock initialized: now 1786343335578837 us; error 40 us; skew 500 ppm
Aug 10 06:28:55 dist-test-slave-45dx krb5kdc[30539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:28:55.581661 628 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:28:55.582585 628 webserver.cc:460] Webserver started at http://127.4.73.3:46403/ using document root <none> and password file <none>
I20260810 06:28:55.583137 628 fs_manager.cc:359] Metadata directory not provided
I20260810 06:28:55.583233 628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:28:55.586971 628 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260810 06:28:55.589072 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:55.590224 628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:28:55.590375 628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "d0e5ae51cabe47dea4e583e3c30ec627"
format_stamp: "Formatted at 2026-08-10 06:28:13 on dist-test-slave-45dx"
server_key: "14f6bf36a40f51e10393c442bf490146"
server_key_iv: "33369568bbe61fcbde87c6e99a7ae7f6"
server_key_version: "encryptionkey@0"
I20260810 06:28:55.590904 628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:28:55.602576 628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:28:55.605396 628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:28:55.605587 628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:28:55.606141 628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:28:55.606925 628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:28:55.606997 628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:55.607067 628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:28:55.607120 628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:28:55.616115 628 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:38029
I20260810 06:28:55.616189 838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:38029 every 8 connection(s)
I20260810 06:28:55.617144 628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:28:55.621002 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.619205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:40493)
0810 06:28:55.620123 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.620225 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":576,"threads_started":1}
W20260810 06:28:55.621497 840 heartbeater.cc:644] Failed to heartbeat to 127.4.73.62:44121 (0 consecutive failures): Network error: Failed to ping master at 127.4.73.62:44121: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
I20260810 06:28:55.621964 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 628
I20260810 06:28:55.622180 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.621928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:33245)
0810 06:28:55.622068 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.622126 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 06:28:55.622815 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:55.622602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:34825)
0810 06:28:55.622702 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:55.622743 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 06:28:55.623003 840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 06:28:56.301440 847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:56.300954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:60509)
0810 06:28:56.301248 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:56.301314 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260810 06:28:56.448915 852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:56.448530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:55299)
0810 06:28:56.448756 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:56.448809 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260810 06:28:56.623904 853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:56.623513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:47857)
0810 06:28:56.623739 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:56.623788 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260810 06:28:56.826995 854 raft_consensus.cc:482] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:28:56.827246 854 raft_consensus.cc:504] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:56.828543 854 leader_election.cc:288] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
Aug 10 06:28:56 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.3@KRBTEST.COM
Aug 10 06:28:56 dist-test-slave-45dx krb5kdc[30539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343335, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.2@KRBTEST.COM
I20260810 06:28:56.843537 856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:56.828784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:38029 (local address 127.4.73.1:48097)
0810 06:28:56.829210 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:56.829224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:56.829292 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:56.831117 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:56.831124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:56.831630 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:56.832219 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:56.832237 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.833701 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.833705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:56.834677 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:56.834687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.835670 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.835674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:56.835874 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:56.836789 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:56.836802 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:56.838458 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:56.840803 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:56.840809 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:56.840823 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:56.841165 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:56.841510 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:56.841513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:56.841515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:56.841706 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:56.842496 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:56.842502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:56.842775 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 06:28:56.843003 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 06:28:56.843330 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":98,"spinlock_wait_cycles":30080,"thread_start_us":87,"threads_started":1}
I20260810 06:28:56.843629 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:56.829113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:48097 (local address 127.4.73.3:38029)
0810 06:28:56.829475 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:56.829497 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:56.829536 (+ 39us) server_negotiation.cc:407] Connection header received
0810 06:28:56.830248 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:56.830255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:56.830644 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:56.830973 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:56.832473 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.833514 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.834821 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.835458 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.835648 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:56.838680 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:56.838686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:56.838698 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:56.838729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:56.840677 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:56.841279 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:56.841283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:56.841287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:56.841409 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:56.841827 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:56.841832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:56.841837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:56.842366 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:56.842509 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:56.843143 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 06:28:56.843447 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260810 06:28:56.844135 859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:56.828965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:41929 (local address 127.4.73.2:44155)
0810 06:28:56.829531 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0810 06:28:56.829552 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 06:28:56.829586 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:28:56.830249 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:28:56.830255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:28:56.830644 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:28:56.830986 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:28:56.832472 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.833513 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.834818 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.835454 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.835640 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:56.839414 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:28:56.839418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:28:56.839429 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:28:56.839458 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:28:56.841546 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:56.842057 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:56.842061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:56.842065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:56.842171 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:28:56.842483 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:28:56.842487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:28:56.842488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:28:56.842915 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:28:56.843039 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 06:28:56.843389 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 06:28:56.843973 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":73,"threads_started":1}
I20260810 06:28:56.844491 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:56.828776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:44155 (local address 127.4.73.1:41929)
0810 06:28:56.829110 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:56.829127 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:28:56.829250 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:28:56.831398 (+ 2148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:28:56.831400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:28:56.831632 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:28:56.832218 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:56.832236 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.833699 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.833702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:56.834674 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:28:56.834686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:28:56.835839 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:28:56.835843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:28:56.835913 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:28:56.836647 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:28:56.836674 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:28:56.839216 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:28:56.841662 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:56.841664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:56.841666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:56.841945 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:56.842266 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:28:56.842270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:28:56.842272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:28:56.842383 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:28:56.843022 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:28:56.843025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:28:56.843123 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 06:28:56.843233 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 06:28:56.843442 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":170,"thread_start_us":92,"threads_started":1}
W20260810 06:28:56.846347 32661 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
W20260810 06:28:56.846501 32658 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
I20260810 06:28:56.846606 32658 leader_election.cc:302] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c; no voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570
I20260810 06:28:56.846980 854 raft_consensus.cc:2726] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:28:57.302640 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:57.302159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:40509)
0810 06:28:57.302456 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:57.302512 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
I20260810 06:28:57.449965 862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:57.449584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:55777)
0810 06:28:57.449823 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:57.449873 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
I20260810 06:28:57.625015 863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:57.624618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:37205)
0810 06:28:57.624848 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:57.624907 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260810 06:28:58.303844 864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:58.303397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:51887)
0810 06:28:58.303686 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:58.303741 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":132,"threads_started":1}
I20260810 06:28:58.451217 865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:58.450732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:40587)
0810 06:28:58.451078 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:58.451135 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":67,"threads_started":1}
I20260810 06:28:58.626147 866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:58.625758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:50565)
0810 06:28:58.626010 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:58.626063 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260810 06:28:58.752991 867 raft_consensus.cc:482] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:28:58.753153 867 raft_consensus.cc:504] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:28:58.753597 867 leader_election.cc:288] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
W20260810 06:28:58.754429 32661 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
W20260810 06:28:58.754552 32658 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
I20260810 06:28:58.754670 32658 leader_election.cc:302] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c; no voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570
I20260810 06:28:58.754845 867 raft_consensus.cc:2726] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:28:59.304978 869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:59.304575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:60711)
0810 06:28:59.304823 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:59.304880 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260810 06:28:59.452416 870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:59.451992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:50855)
0810 06:28:59.452243 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:59.452322 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260810 06:28:59.627328 872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:28:59.626907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:58801)
0810 06:28:59.627183 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 06:28:59.627237 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260810 06:29:00.306305 875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:00.305849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:40655)
0810 06:29:00.306144 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:00.306207 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
I20260810 06:29:00.453478 876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:00.453103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:33299)
0810 06:29:00.453339 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:00.453397 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260810 06:29:00.628684 878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:00.628223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:46485)
0810 06:29:00.628543 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:00.628596 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":160,"threads_started":1}
I20260810 06:29:01.307492 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:01.307091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:49187)
0810 06:29:01.307339 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:01.307395 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260810 06:29:01.454560 880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:01.454201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:59035)
0810 06:29:01.454433 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:01.454481 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260810 06:29:01.629830 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:01.629432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:34997)
0810 06:29:01.629680 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:01.629736 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260810 06:29:01.677863 885 raft_consensus.cc:482] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:01.678011 885 raft_consensus.cc:504] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:29:01.678503 885 leader_election.cc:288] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
W20260810 06:29:01.679222 32661 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
W20260810 06:29:01.679324 32658 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
I20260810 06:29:01.679442 32658 leader_election.cc:302] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c; no voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570
I20260810 06:29:01.679592 885 raft_consensus.cc:2726] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:29:02.308755 886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:02.308249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:55577)
0810 06:29:02.308575 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:02.308632 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":132,"threads_started":1}
I20260810 06:29:02.460903 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:02.460446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:44381)
0810 06:29:02.460742 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:02.460804 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260810 06:29:02.631134 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:02.630627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:42255)
0810 06:29:02.630927 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:02.631013 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":98,"threads_started":1}
I20260810 06:29:03.309867 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:03.309448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:50429)
0810 06:29:03.309712 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:03.309767 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":87,"threads_started":1}
I20260810 06:29:03.462090 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:03.461654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:36999)
0810 06:29:03.461933 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:03.461990 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
I20260810 06:29:03.632455 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:03.631968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:33465)
0810 06:29:03.632237 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:03.632296 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260810 06:29:04.311117 896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:04.310605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:44759)
0810 06:29:04.310954 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:04.311022 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":84,"threads_started":1}
I20260810 06:29:04.463481 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:04.463028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:39739)
0810 06:29:04.463324 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:04.463393 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":105,"threads_started":1}
I20260810 06:29:04.633544 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:04.633163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:44259)
0810 06:29:04.633381 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:04.633431 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260810 06:29:04.770949 903 raft_consensus.cc:482] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:04.771085 903 raft_consensus.cc:504] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:29:04.771485 903 leader_election.cc:288] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
W20260810 06:29:04.772282 32658 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
W20260810 06:29:04.772418 32661 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
I20260810 06:29:04.772607 32661 leader_election.cc:302] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c; no voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570
I20260810 06:29:04.772773 903 raft_consensus.cc:2726] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:29:05.312623 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:05.311954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:49435)
0810 06:29:05.312215 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:05.312266 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260810 06:29:05.464692 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:05.464239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:41431)
0810 06:29:05.464545 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:05.464600 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":143,"threads_started":1}
I20260810 06:29:05.634703 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:05.634288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:51851)
0810 06:29:05.634552 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:05.634610 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260810 06:29:06.313735 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:06.313310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:39187)
0810 06:29:06.313585 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:06.313639 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260810 06:29:06.465881 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:06.465438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:51267)
0810 06:29:06.465706 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:06.465763 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260810 06:29:06.635864 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:06.635454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:60665)
0810 06:29:06.635707 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:06.635762 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260810 06:29:07.314802 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:07.314429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:38155)
0810 06:29:07.314661 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:07.314711 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260810 06:29:07.467075 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:07.466658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:34249)
0810 06:29:07.466927 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:07.466982 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260810 06:29:07.637079 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:07.636661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:50091)
0810 06:29:07.636930 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:07.636985 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260810 06:29:08.315927 921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:08.315516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:53825)
0810 06:29:08.315756 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:08.315809 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260810 06:29:08.468384 922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:08.467811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:41247)
0810 06:29:08.468184 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:08.468241 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":74,"threads_started":1}
I20260810 06:29:08.638250 923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:08.637871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:36841)
0810 06:29:08.638111 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:08.638161 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260810 06:29:09.317108 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:09.316708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:54633)
0810 06:29:09.316956 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:09.317011 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260810 06:29:09.469765 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:09.469328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:42575)
0810 06:29:09.469612 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:09.469669 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260810 06:29:09.618153 929 raft_consensus.cc:482] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:09.618297 929 raft_consensus.cc:504] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0e5ae51cabe47dea4e583e3c30ec627" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 38029 } } peers { permanent_uuid: "d55186e037a947418cf0b19a5d455570" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 44155 } } peers { permanent_uuid: "11f48fb346c74bf6980b680bb874785c" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40933 } }
I20260810 06:29:09.618732 929 leader_election.cc:288] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029), d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155)
W20260810 06:29:09.619503 32661 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d0e5ae51cabe47dea4e583e3c30ec627 (127.4.73.3:38029): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
W20260810 06:29:09.619614 32658 leader_election.cc:341] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d55186e037a947418cf0b19a5d455570 (127.4.73.2:44155): Not found: Tablet not found: 1e005e56a9714dac9c8687c81122b4ad
I20260810 06:29:09.619707 32658 leader_election.cc:302] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 11f48fb346c74bf6980b680bb874785c; no voters: d0e5ae51cabe47dea4e583e3c30ec627, d55186e037a947418cf0b19a5d455570
I20260810 06:29:09.619853 929 raft_consensus.cc:2726] T 1e005e56a9714dac9c8687c81122b4ad P 11f48fb346c74bf6980b680bb874785c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 06:29:09.639408 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:09.639020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:43223)
0810 06:29:09.639264 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:09.639314 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260810 06:29:10.318457 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:10.318038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.1:56977)
0810 06:29:10.318304 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:10.318360 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":110,"threads_started":1}
I20260810 06:29:10.471040 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:10.470633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.2:56609)
0810 06:29:10.470892 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:10.470946 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260810 06:29:10.629123 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 32568
I20260810 06:29:10.640950 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:10.640465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:44121 (local address 127.4.73.3:49053)
0810 06:29:10.640797 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:10.640855 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.62:44121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260810 06:29:10.644842 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 393
I20260810 06:29:10.657574 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 628
2026-08-10T06:29:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59798 ms)
[----------] 30 tests from SecurityITest (254073 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:10 dist-test-slave-45dx krb5kdc[938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:10 dist-test-slave-45dx krb5kdc[938](info): set up 2 sockets
Aug 10 06:29:10 dist-test-slave-45dx krb5kdc[938](info): commencing operation
krb5kdc: starting...
W20260810 06:29:11.921088 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.239s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:12.044878 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:41361
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:41361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:12.158286 955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:12.158524 955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:12.158560 955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:12.162231 955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:12.162281 955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:12.162299 955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:12.162357 955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:12.162386 955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:12.167179 955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:41361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:41361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:12.168133 955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:12.169119 955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:12.177559 962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:12.177696 955 server_base.cc:1034] running on GCE node
W20260810 06:29:12.177534 965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:12.177551 968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:12.178516 955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:12.179543 955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:12.180756 955 hybrid_clock.cc:648] HybridClock initialized: now 1786343352180741 us; error 30 us; skew 500 ppm
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:12.183544 955 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:12.184487 955 webserver.cc:460] Webserver started at http://127.4.73.62:46081/ using document root <none> and password file <none>
I20260810 06:29:12.185040 955 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:12.185134 955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:12.185374 955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:12.187321 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "5c904028433b49aaa865332df1311765"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "4582f7967a1695557ddccd2820c4a7bf"
server_key_iv: "5b7d0d59ceb7799508b13163850901a4"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.187894 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "5c904028433b49aaa865332df1311765"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "4582f7967a1695557ddccd2820c4a7bf"
server_key_iv: "5b7d0d59ceb7799508b13163850901a4"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.191534 955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 06:29:12.193872 981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.194960 955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:29:12.195111 955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "5c904028433b49aaa865332df1311765"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "4582f7967a1695557ddccd2820c4a7bf"
server_key_iv: "5b7d0d59ceb7799508b13163850901a4"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.195222 955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:12.209942 955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:12.212836 955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:12.213042 955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:12.219884 955 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:41361
I20260810 06:29:12.219955 1067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:41361 every 8 connection(s)
I20260810 06:29:12.220868 955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:12.221419 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 955
I20260810 06:29:12.221560 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:12.221858 4388 external_mini_cluster.cc:1428] Setting key 6fa8ddbc503cbf7f57f6e7020aee8d95
I20260810 06:29:12.224133 1069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:12.230942 1069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765: Bootstrap starting.
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:12.233633 1069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:12.234514 1069 log.cc:824] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:12.236732 1069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765: No bootstrap required, opened a new log
I20260810 06:29:12.237037 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.223242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.4.73.62:41361)
0810 06:29:12.223684 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.223702 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.223740 (+ 38us) server_negotiation.cc:407] Connection header received
0810 06:29:12.224627 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.224649 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.225038 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.225373 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:12.226073 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.227604 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.228519 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.229253 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.229406 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.231991 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:12.232000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:12.232016 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:12.232046 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:12.234063 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.234473 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.234479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.234485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.234632 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.234903 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.234907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.234909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.235370 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:12.235553 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.235804 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.236090 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260810 06:29:12.240595 1069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c904028433b49aaa865332df1311765" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41361 } }
I20260810 06:29:12.240805 1069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:12.240849 1069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c904028433b49aaa865332df1311765, State: Initialized, Role: FOLLOWER
I20260810 06:29:12.241237 1069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c904028433b49aaa865332df1311765" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41361 } }
I20260810 06:29:12.241367 1069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:12.241415 1069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:12.241483 1069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:12.242309 1069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c904028433b49aaa865332df1311765" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41361 } }
I20260810 06:29:12.242609 1069 leader_election.cc:302] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c904028433b49aaa865332df1311765; no voters:
I20260810 06:29:12.242849 1069 leader_election.cc:288] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:12.242995 1080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:12.243259 1080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [term 1 LEADER]: Becoming Leader. State: Replica: 5c904028433b49aaa865332df1311765, State: Running, Role: LEADER
I20260810 06:29:12.243583 1080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c904028433b49aaa865332df1311765" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41361 } }
I20260810 06:29:12.244166 1069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:12.245339 1081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c904028433b49aaa865332df1311765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c904028433b49aaa865332df1311765" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41361 } } }
I20260810 06:29:12.245504 1081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:12.245992 1092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:12.245918 1083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c904028433b49aaa865332df1311765. Latest consensus state: current_term: 1 leader_uuid: "5c904028433b49aaa865332df1311765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c904028433b49aaa865332df1311765" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41361 } } }
I20260810 06:29:12.246511 1083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:12.248958 1092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:12.255084 1092 catalog_manager.cc:1347] Generated new cluster ID: 3420a23e640c416e9741f07a396380ac
I20260810 06:29:12.255152 1092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:12.277215 1092 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:12.278146 1092 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:12.282841 1092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765: Generated new TSK 0
I20260810 06:29:12.283476 1092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:29:12.319067 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:41361
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:12.430634 1115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:12.431079 1115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:12.431154 1115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:12.435144 1115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:12.435189 1115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:12.435272 1115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:29:12.440183 1115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:41361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:12.441148 1115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:12.442140 1115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:12.451429 1123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:12.451442 1124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:12.451460 1128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:12.451800 1115 server_base.cc:1034] running on GCE node
I20260810 06:29:12.452432 1115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:12.453043 1115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:12.454244 1115 hybrid_clock.cc:648] HybridClock initialized: now 1786343352454223 us; error 42 us; skew 500 ppm
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:12.456961 1115 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:29:12.457702 1115 webserver.cc:460] Webserver started at http://127.4.73.1:45843/ using document root <none> and password file <none>
I20260810 06:29:12.458202 1115 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:12.458257 1115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:12.458406 1115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:12.460232 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "1a1ccc890fc34347902bfca494ddbbf2"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "c77be641f040d4b68a69e1e14526f477"
server_key_iv: "16648783c34fcc2a0e8eb21d9a88a9bb"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.460803 1115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "1a1ccc890fc34347902bfca494ddbbf2"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "c77be641f040d4b68a69e1e14526f477"
server_key_iv: "16648783c34fcc2a0e8eb21d9a88a9bb"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.464110 1115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:29:12.466377 1138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.467540 1115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:29:12.467634 1115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "1a1ccc890fc34347902bfca494ddbbf2"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "c77be641f040d4b68a69e1e14526f477"
server_key_iv: "16648783c34fcc2a0e8eb21d9a88a9bb"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.467713 1115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:12.479480 1115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:12.483026 1115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:12.483182 1115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:12.483731 1115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:12.484602 1115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:12.484651 1115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.484691 1115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:12.484723 1115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.494007 1115 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:42901
I20260810 06:29:12.494045 1341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:42901 every 8 connection(s)
I20260810 06:29:12.494905 1115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:29:12.495726 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 1115
I20260810 06:29:12.495872 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:29:12.496153 4388 external_mini_cluster.cc:1428] Setting key ed51cc6bda6afe9ca043cbcb6f0cde5d
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:29:12.510190 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.496900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:34869 (local address 127.4.73.62:41361)
0810 06:29:12.497059 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.497062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.498076 (+ 1014us) server_negotiation.cc:407] Connection header received
0810 06:29:12.499618 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.499624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.499693 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.499833 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:12.501595 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.502490 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.503639 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.504052 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.504182 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.506644 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:12.506650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:12.506654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:12.506688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:12.508220 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.508796 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.508799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.508800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.508884 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.509201 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.509203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.509204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.509382 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:12.509454 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.509936 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.510069 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:29:12.511075 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.497364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:41361 (local address 127.4.73.1:34869)
0810 06:29:12.497892 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:12.497947 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:12.499417 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:12.500073 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:12.500082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:12.500699 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:12.501417 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.501442 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.502631 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.502634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.503516 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.503525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.504216 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.504219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.504423 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.505014 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:12.505030 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:12.506466 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:12.508384 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:12.508389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:12.508399 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:12.508693 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:12.508969 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:12.508971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:12.508972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:12.509112 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:12.509468 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:12.509473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:12.509692 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 06:29:12.509835 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 06:29:12.510080 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":157,"threads_started":1}
I20260810 06:29:12.512457 1342 heartbeater.cc:347] Connected to a master server at 127.4.73.62:41361
I20260810 06:29:12.512750 1342 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:12.513370 1342 heartbeater.cc:511] Master 127.4.73.62:41361 requested a full tablet report, sending...
I20260810 06:29:12.515027 1014 ts_manager.cc:194] Registered new tserver with Master: 1a1ccc890fc34347902bfca494ddbbf2 (127.4.73.1:42901)
I20260810 06:29:12.516217 1014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:34869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:29:12.528122 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:41361
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:12.639503 1354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:12.639781 1354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:12.639863 1354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:12.644147 1354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:12.644228 1354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:12.644343 1354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:29:12.649829 1354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:41361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:12.650740 1354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:12.651672 1354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:12.660836 1371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:12.660836 1363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:12.660925 1364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:12.660904 1354 server_base.cc:1034] running on GCE node
I20260810 06:29:12.661471 1354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:12.662058 1354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:12.663244 1354 hybrid_clock.cc:648] HybridClock initialized: now 1786343352663234 us; error 30 us; skew 500 ppm
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:12.670186 1354 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:29:12.670982 1354 webserver.cc:460] Webserver started at http://127.4.73.2:42469/ using document root <none> and password file <none>
I20260810 06:29:12.671593 1354 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:12.671658 1354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:12.671885 1354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:12.673978 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "d3d35262fc73458588112a0344eafa9a"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "65dc9344b2804304e4ad0b97ee0c8693"
server_key_iv: "22904c18f99d6f8710dcbf298ba304a5"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.674530 1354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "d3d35262fc73458588112a0344eafa9a"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "65dc9344b2804304e4ad0b97ee0c8693"
server_key_iv: "22904c18f99d6f8710dcbf298ba304a5"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.678205 1354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:29:12.680601 1379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.681795 1354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:29:12.681922 1354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "d3d35262fc73458588112a0344eafa9a"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "65dc9344b2804304e4ad0b97ee0c8693"
server_key_iv: "22904c18f99d6f8710dcbf298ba304a5"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.682027 1354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:12.708062 1354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:12.711324 1354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:12.711561 1354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:12.712153 1354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:12.713054 1354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:12.713128 1354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.713203 1354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:12.713246 1354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.722124 1354 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:43687
I20260810 06:29:12.722146 1577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:43687 every 8 connection(s)
I20260810 06:29:12.723018 1354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:29:12.725560 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 1354
I20260810 06:29:12.725710 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:29:12.726066 4388 external_mini_cluster.cc:1428] Setting key 4ff6b96e98aa692ece8721bdc426acb9
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:29:12.737881 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.724964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:43309 (local address 127.4.73.62:41361)
0810 06:29:12.725130 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.725135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.726028 (+ 893us) server_negotiation.cc:407] Connection header received
0810 06:29:12.727576 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.727580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.727636 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.727741 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:12.729001 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.729652 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.730833 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.731099 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.731183 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.733999 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:12.734005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:12.734010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:12.734041 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:12.735752 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.736264 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.736269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.736271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.736424 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.736800 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.736803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.736805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.736991 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:12.737103 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.737584 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.737708 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 06:29:12.738611 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.725383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:41361 (local address 127.4.73.2:43309)
0810 06:29:12.725847 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:12.725911 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:12.727341 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:12.727933 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:12.727951 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:12.728370 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:12.728848 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.728862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.729809 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.729812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.730701 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.730711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.731260 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.731266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.731500 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.732140 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:12.732165 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:12.733829 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:12.735873 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:12.735878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:12.735888 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:12.736150 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:12.736522 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:12.736527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:12.736530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:12.736668 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:12.737103 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:12.737110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:12.737343 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 06:29:12.737527 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 06:29:12.737836 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":130,"threads_started":1}
I20260810 06:29:12.739655 1579 heartbeater.cc:347] Connected to a master server at 127.4.73.62:41361
I20260810 06:29:12.739971 1579 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
I20260810 06:29:12.740566 1579 heartbeater.cc:511] Master 127.4.73.62:41361 requested a full tablet report, sending...
I20260810 06:29:12.741657 1014 ts_manager.cc:194] Registered new tserver with Master: d3d35262fc73458588112a0344eafa9a (127.4.73.2:43687)
I20260810 06:29:12.742293 1014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:43309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:29:12.755167 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:41361
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:12.863384 1586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:12.863613 1586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:12.863652 1586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:12.867625 1586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:12.867686 1586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:12.867780 1586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:29:12.873085 1586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:41361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:12.874104 1586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:12.875102 1586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:12.884822 1597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:12.884889 1601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:12.884889 1596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:12.885042 1586 server_base.cc:1034] running on GCE node
I20260810 06:29:12.885437 1586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:12.886068 1586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:12.887255 1586 hybrid_clock.cc:648] HybridClock initialized: now 1786343352887237 us; error 34 us; skew 500 ppm
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:12.890012 1586 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:29:12.890805 1586 webserver.cc:460] Webserver started at http://127.4.73.3:39635/ using document root <none> and password file <none>
I20260810 06:29:12.891328 1586 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:12.891422 1586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:12.891644 1586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:12.893630 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "eb139bf5479e448299bf30acd2901afe"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "78715d0c8a6b196026d0b60868c05f8a"
server_key_iv: "2b56fbdee729a73d74206c3da93e8f7d"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.894183 1586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "eb139bf5479e448299bf30acd2901afe"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "78715d0c8a6b196026d0b60868c05f8a"
server_key_iv: "2b56fbdee729a73d74206c3da93e8f7d"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.897644 1586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:12.899857 1614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.901077 1586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:12.901211 1586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "eb139bf5479e448299bf30acd2901afe"
format_stamp: "Formatted at 2026-08-10 06:29:12 on dist-test-slave-45dx"
server_key: "78715d0c8a6b196026d0b60868c05f8a"
server_key_iv: "2b56fbdee729a73d74206c3da93e8f7d"
server_key_version: "encryptionkey@0"
I20260810 06:29:12.901317 1586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:12.914788 1586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:12.918006 1586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:12.918203 1586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:12.918792 1586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:12.919596 1586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:12.919668 1586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.919740 1586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:12.919792 1586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:12.928535 1586 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:42329
I20260810 06:29:12.928568 1799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:42329 every 8 connection(s)
I20260810 06:29:12.929335 1586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:29:12.931047 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 1586
I20260810 06:29:12.931185 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:29:12.931496 4388 external_mini_cluster.cc:1428] Setting key 525b7726a041334a0cfa9c2242ea75a0
Aug 10 06:29:12 dist-test-slave-45dx krb5kdc[938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343352, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:12.942179 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.930899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:57663 (local address 127.4.73.62:41361)
0810 06:29:12.931032 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.931043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.932031 (+ 988us) server_negotiation.cc:407] Connection header received
0810 06:29:12.933656 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.933661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.933716 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.933797 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:12.934888 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.935432 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.936612 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.936836 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.936921 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.939176 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:12.939179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:12.939181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:12.939203 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:12.940519 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.940972 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.940975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.940976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.941044 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.941313 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.941315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.941316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.941473 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:12.941561 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.941981 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.942070 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:29:12.942899 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.931332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:41361 (local address 127.4.73.3:57663)
0810 06:29:12.931885 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:12.931943 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:12.933402 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:12.933968 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:12.933974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:12.934299 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:12.934756 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.934769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.935541 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.935543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.936508 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.936518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.936964 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.936967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.937102 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.937617 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:12.937632 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:12.939032 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:12.940633 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:12.940637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:12.940646 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:12.940893 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:12.941118 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:12.941120 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:12.941122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:12.941241 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:12.941573 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:12.941577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:12.941786 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 06:29:12.941932 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 06:29:12.942151 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":138,"threads_started":1}
I20260810 06:29:12.944046 1800 heartbeater.cc:347] Connected to a master server at 127.4.73.62:41361
I20260810 06:29:12.944295 1800 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:12.944836 1800 heartbeater.cc:511] Master 127.4.73.62:41361 requested a full tablet report, sending...
I20260810 06:29:12.945935 1014 ts_manager.cc:194] Registered new tserver with Master: eb139bf5479e448299bf30acd2901afe (127.4.73.3:42329)
I20260810 06:29:12.946475 1014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:57663
I20260810 06:29:12.957089 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:29:12.965715 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.958992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.4.73.62:41361)
0810 06:29:12.959101 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.959104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.959327 (+ 223us) server_negotiation.cc:407] Connection header received
0810 06:29:12.959424 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.959427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.959461 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.959528 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:12.960218 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.960774 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.961553 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.961755 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.961828 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.962994 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:12.962997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:12.962999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:12.963020 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:12.964180 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.964606 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.964609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.964610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.964683 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:12.964917 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:12.964919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:12.964920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:12.965122 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:12.965224 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.965456 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.965566 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:29:12.968266 1014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:29:12.971151 1014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:29:12.985571 1817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.979609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:42329 (local address 127.0.0.1:39050)
0810 06:29:12.980523 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:12.980535 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:12.980605 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:12.981726 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:12.981730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:12.981762 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:12.981889 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.981895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.983160 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.983163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.984110 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.984120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.985014 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.985018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.985120 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.985160 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 06:29:12.985285 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 06:29:12.985420 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":33,"thread_start_us":88,"threads_started":1}
I20260810 06:29:12.985630 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.979965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.4.73.3:42329)
0810 06:29:12.981289 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.981299 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.981320 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:29:12.981395 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.981400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.981576 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.981726 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:12.982018 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.983016 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.984236 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.984839 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.984985 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.985134 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.985394 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.985489 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1199,"thread_start_us":101,"threads_started":1}
I20260810 06:29:12.985920 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.979344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.4.73.1:42901)
0810 06:29:12.979704 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.979719 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.980135 (+ 416us) server_negotiation.cc:407] Connection header received
0810 06:29:12.980474 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.980481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.980681 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.980800 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:12.981395 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.982377 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.984212 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.985084 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.985192 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.985355 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.985667 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.985781 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":64,"threads_started":1}
I20260810 06:29:12.986203 1821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.979385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:43687 (local address 127.0.0.1:60398)
0810 06:29:12.980105 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:12.980120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:12.980257 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:12.980968 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:12.980973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:12.980993 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:12.981164 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.981172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.982652 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.982655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.984089 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.984096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.985347 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.985350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.985444 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.985467 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:29:12.985517 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 06:29:12.985594 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":150,"spinlock_wait_cycles":8704,"thread_start_us":101,"threads_started":1}
I20260810 06:29:12.986528 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.979190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:42901 (local address 127.0.0.1:46444)
0810 06:29:12.979986 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:12.980037 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:12.980259 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:12.981025 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:12.981028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:12.981045 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:12.981221 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.981229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.982503 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.982510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.984075 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:12.984087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.985439 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.985443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:12.985525 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.985541 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 06:29:12.986226 (+ 685us) client_negotiation.cc:240] Negotiation successful
0810 06:29:12.986307 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":160,"spinlock_wait_cycles":7040,"thread_start_us":70,"threads_started":1}
I20260810 06:29:12.987031 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:12.979974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.4.73.2:43687)
0810 06:29:12.980554 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:12.980569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:12.980589 (+ 20us) server_negotiation.cc:407] Connection header received
0810 06:29:12.980652 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:12.980658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:12.980808 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:12.980952 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:12.981294 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.982513 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.984404 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:12.985212 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:12.986559 (+ 1347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:12.986708 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:12.986780 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 06:29:12.986886 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":123,"threads_started":1}
I20260810 06:29:12.988886 1699 tablet_service.cc:1467] Processing CreateTablet for tablet d847378e786c44cdbdd5a483e802573d (DEFAULT_TABLE table=test-table [id=3cc068af0bb6488687fc7ec21950d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:12.989164 1473 tablet_service.cc:1467] Processing CreateTablet for tablet d847378e786c44cdbdd5a483e802573d (DEFAULT_TABLE table=test-table [id=3cc068af0bb6488687fc7ec21950d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:12.989254 1235 tablet_service.cc:1467] Processing CreateTablet for tablet d847378e786c44cdbdd5a483e802573d (DEFAULT_TABLE table=test-table [id=3cc068af0bb6488687fc7ec21950d06f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:12.990036 1699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d847378e786c44cdbdd5a483e802573d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:12.990080 1473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d847378e786c44cdbdd5a483e802573d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:12.990196 1235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d847378e786c44cdbdd5a483e802573d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:12.995615 1824 tablet_bootstrap.cc:492] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Bootstrap starting.
I20260810 06:29:12.996984 1826 tablet_bootstrap.cc:492] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Bootstrap starting.
I20260810 06:29:12.997613 1827 tablet_bootstrap.cc:492] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2: Bootstrap starting.
I20260810 06:29:12.998603 1824 tablet_bootstrap.cc:654] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:12.999097 1826 tablet_bootstrap.cc:654] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:12.999534 1827 tablet_bootstrap.cc:654] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:12.999527 1824 log.cc:824] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:13.000293 1826 log.cc:824] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:13.000826 1827 log.cc:824] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:13.001931 1824 tablet_bootstrap.cc:492] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: No bootstrap required, opened a new log
I20260810 06:29:13.002130 1824 ts_tablet_manager.cc:1397] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 06:29:13.002732 1827 tablet_bootstrap.cc:492] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2: No bootstrap required, opened a new log
I20260810 06:29:13.002942 1827 ts_tablet_manager.cc:1397] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:29:13.002964 1826 tablet_bootstrap.cc:492] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: No bootstrap required, opened a new log
I20260810 06:29:13.003129 1826 ts_tablet_manager.cc:1397] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:29:13.006148 1827 raft_consensus.cc:348] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.006163 1824 raft_consensus.cc:348] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.006409 1827 raft_consensus.cc:374] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:13.006453 1824 raft_consensus.cc:374] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:13.006520 1824 raft_consensus.cc:729] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d35262fc73458588112a0344eafa9a, State: Initialized, Role: FOLLOWER
I20260810 06:29:13.006459 1827 raft_consensus.cc:729] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1ccc890fc34347902bfca494ddbbf2, State: Initialized, Role: FOLLOWER
I20260810 06:29:13.006918 1826 raft_consensus.cc:348] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.007210 1826 raft_consensus.cc:374] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:13.007151 1827 consensus_queue.cc:260] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.007272 1826 raft_consensus.cc:729] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb139bf5479e448299bf30acd2901afe, State: Initialized, Role: FOLLOWER
I20260810 06:29:13.007782 1826 consensus_queue.cc:260] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.008044 1824 consensus_queue.cc:260] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.008279 1827 ts_tablet_manager.cc:1428] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:29:13.008710 1342 heartbeater.cc:503] Master 127.4.73.62:41361 was elected leader, sending a full tablet report...
I20260810 06:29:13.008723 1579 heartbeater.cc:503] Master 127.4.73.62:41361 was elected leader, sending a full tablet report...
I20260810 06:29:13.008771 1826 ts_tablet_manager.cc:1428] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:29:13.008920 1824 ts_tablet_manager.cc:1428] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 06:29:13.009712 1800 heartbeater.cc:503] Master 127.4.73.62:41361 was elected leader, sending a full tablet report...
I20260810 06:29:13.070689 1833 raft_consensus.cc:482] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:13.070922 1833 raft_consensus.cc:504] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.072181 1833 leader_election.cc:288] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a1ccc890fc34347902bfca494ddbbf2 (127.4.73.1:42901), d3d35262fc73458588112a0344eafa9a (127.4.73.2:43687)
I20260810 06:29:13.077109 1819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.072641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:42075 (local address 127.4.73.1:42901)
0810 06:29:13.072846 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:13.072851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:13.072872 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:29:13.072947 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:13.072951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:13.073017 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:13.073111 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:13.073465 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.074339 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.075836 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.076495 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.076609 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.076696 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:13.076901 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 06:29:13.076981 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 06:29:13.077109 1823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.072758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:52047 (local address 127.4.73.2:43687)
0810 06:29:13.072924 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:13.072929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:13.073337 (+ 408us) server_negotiation.cc:407] Connection header received
0810 06:29:13.073472 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:13.073475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:13.073529 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:13.073612 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:13.073855 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.074546 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.075944 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.076588 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.076680 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.076743 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:13.076901 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 06:29:13.076974 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 06:29:13.077142 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.072506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:42901 (local address 127.4.73.3:42075)
0810 06:29:13.072658 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:13.072676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:13.072814 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:13.073141 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:13.073145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:13.073168 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:13.073309 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.073318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.074447 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.074449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.075693 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.075704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.076616 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.076619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.076756 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.076782 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 06:29:13.076865 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 06:29:13.076979 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":25}
I20260810 06:29:13.077163 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.072506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:43687 (local address 127.4.73.3:52047)
0810 06:29:13.073233 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:13.073247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:13.073345 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:13.073630 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:13.073632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:13.073648 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:13.073740 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.073744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.074657 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.074660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.075816 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.075824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.076688 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.076691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.076769 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.076784 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 06:29:13.076865 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 06:29:13.076979 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":336,"thread_start_us":152,"threads_started":1}
I20260810 06:29:13.077849 1265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d847378e786c44cdbdd5a483e802573d" candidate_uuid: "eb139bf5479e448299bf30acd2901afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1ccc890fc34347902bfca494ddbbf2" is_pre_election: true
I20260810 06:29:13.077936 1508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d847378e786c44cdbdd5a483e802573d" candidate_uuid: "eb139bf5479e448299bf30acd2901afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d35262fc73458588112a0344eafa9a" is_pre_election: true
I20260810 06:29:13.078186 1265 raft_consensus.cc:2454] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb139bf5479e448299bf30acd2901afe in term 0.
I20260810 06:29:13.078272 1508 raft_consensus.cc:2454] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb139bf5479e448299bf30acd2901afe in term 0.
I20260810 06:29:13.078737 1620 leader_election.cc:302] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a1ccc890fc34347902bfca494ddbbf2, eb139bf5479e448299bf30acd2901afe; no voters:
I20260810 06:29:13.079042 1833 raft_consensus.cc:2781] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:29:13.079149 1833 raft_consensus.cc:482] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:29:13.079210 1833 raft_consensus.cc:3037] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:13.080264 1833 raft_consensus.cc:504] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.080672 1833 leader_election.cc:288] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [CANDIDATE]: Term 1 election: Requested vote from peers 1a1ccc890fc34347902bfca494ddbbf2 (127.4.73.1:42901), d3d35262fc73458588112a0344eafa9a (127.4.73.2:43687)
I20260810 06:29:13.081066 1508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d847378e786c44cdbdd5a483e802573d" candidate_uuid: "eb139bf5479e448299bf30acd2901afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d35262fc73458588112a0344eafa9a"
I20260810 06:29:13.081171 1265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d847378e786c44cdbdd5a483e802573d" candidate_uuid: "eb139bf5479e448299bf30acd2901afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a1ccc890fc34347902bfca494ddbbf2"
I20260810 06:29:13.081204 1508 raft_consensus.cc:3037] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:13.081286 1265 raft_consensus.cc:3037] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:13.082506 1508 raft_consensus.cc:2454] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb139bf5479e448299bf30acd2901afe in term 1.
I20260810 06:29:13.082609 1265 raft_consensus.cc:2454] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb139bf5479e448299bf30acd2901afe in term 1.
I20260810 06:29:13.082818 1619 leader_election.cc:302] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3d35262fc73458588112a0344eafa9a, eb139bf5479e448299bf30acd2901afe; no voters:
I20260810 06:29:13.083006 1833 raft_consensus.cc:2781] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:13.083220 1833 raft_consensus.cc:686] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 1 LEADER]: Becoming Leader. State: Replica: eb139bf5479e448299bf30acd2901afe, State: Running, Role: LEADER
I20260810 06:29:13.083541 1833 consensus_queue.cc:237] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } }
I20260810 06:29:13.087007 1014 catalog_manager.cc:5495] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe reported cstate change: term changed from 0 to 1, leader changed from <none> to eb139bf5479e448299bf30acd2901afe (127.4.73.3). New cstate: current_term: 1 leader_uuid: "eb139bf5479e448299bf30acd2901afe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb139bf5479e448299bf30acd2901afe" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42329 } health_report { overall_health: HEALTHY } } }
I20260810 06:29:13.113894 1822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.110492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.4.73.3:42329)
0810 06:29:13.110625 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:13.110629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:13.110643 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:29:13.110762 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:13.110765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:13.110819 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:13.110911 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:13.111180 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.111708 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.112544 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.112762 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.112845 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.113113 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:13.113546 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:13.113650 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:13.113709 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 06:29:13.113760 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:29:13.119890 1265 raft_consensus.cc:1261] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 1 FOLLOWER]: Refusing update from remote peer eb139bf5479e448299bf30acd2901afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:13.119890 1508 raft_consensus.cc:1261] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 1 FOLLOWER]: Refusing update from remote peer eb139bf5479e448299bf30acd2901afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:13.120682 1833 consensus_queue.cc:1035] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d35262fc73458588112a0344eafa9a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:13.120918 1839 consensus_queue.cc:1035] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a1ccc890fc34347902bfca494ddbbf2" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:13.128825 1849 mvcc.cc:205] Tried to move back new op lower bound from 7316862374371782656 to 7316862374233059328. Current Snapshot: MvccSnapshot[applied={T|T < 7316862374371782656}]
I20260810 06:29:13.132112 1850 mvcc.cc:205] Tried to move back new op lower bound from 7316862374371782656 to 7316862374233059328. Current Snapshot: MvccSnapshot[applied={T|T < 7316862374371782656 or (T in {7316862374371782656})}]
I20260810 06:29:13.137487 1014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:13.137696 1014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:13.140251 1014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c904028433b49aaa865332df1311765: Sending DeleteTablet for 3 replicas of tablet d847378e786c44cdbdd5a483e802573d
I20260810 06:29:13.141129 1235 tablet_service.cc:1514] Processing DeleteTablet for tablet d847378e786c44cdbdd5a483e802573d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:13 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:46444
I20260810 06:29:13.141134 1699 tablet_service.cc:1514] Processing DeleteTablet for tablet d847378e786c44cdbdd5a483e802573d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:13 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:39050
I20260810 06:29:13.141664 1866 tablet_replica.cc:330] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: stopping tablet replica
I20260810 06:29:13.141688 1473 tablet_service.cc:1514] Processing DeleteTablet for tablet d847378e786c44cdbdd5a483e802573d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:13 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:60398
I20260810 06:29:13.142014 1866 raft_consensus.cc:2229] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:29:13.142261 1871 tablet_replica.cc:330] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: stopping tablet replica
I20260810 06:29:13.142428 1866 raft_consensus.cc:2258] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:13.142679 1871 raft_consensus.cc:2229] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:13.142812 1868 tablet_replica.cc:330] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2: stopping tablet replica
I20260810 06:29:13.143046 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 1115
I20260810 06:29:13.143115 1868 raft_consensus.cc:2229] T d847378e786c44cdbdd5a483e802573d P 1a1ccc890fc34347902bfca494ddbbf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:13.145947 1871 raft_consensus.cc:2258] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:13.162456 1866 ts_tablet_manager.cc:1905] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:13.164641 1871 ts_tablet_manager.cc:1905] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:13.165544 1866 ts_tablet_manager.cc:1918] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:13.165658 1866 log.cc:1196] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/d847378e786c44cdbdd5a483e802573d
I20260810 06:29:13.166029 1866 ts_tablet_manager.cc:1939] T d847378e786c44cdbdd5a483e802573d P eb139bf5479e448299bf30acd2901afe: Deleting consensus metadata
I20260810 06:29:13.167326 983 catalog_manager.cc:4841] TS eb139bf5479e448299bf30acd2901afe (127.4.73.3:42329): tablet d847378e786c44cdbdd5a483e802573d (table test-table [id=3cc068af0bb6488687fc7ec21950d06f]) successfully deleted
I20260810 06:29:13.167879 1871 ts_tablet_manager.cc:1918] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:13.167999 1871 log.cc:1196] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/d847378e786c44cdbdd5a483e802573d
I20260810 06:29:13.168390 1871 ts_tablet_manager.cc:1939] T d847378e786c44cdbdd5a483e802573d P d3d35262fc73458588112a0344eafa9a: Deleting consensus metadata
I20260810 06:29:13.169409 985 catalog_manager.cc:4841] TS d3d35262fc73458588112a0344eafa9a (127.4.73.2:43687): tablet d847378e786c44cdbdd5a483e802573d (table test-table [id=3cc068af0bb6488687fc7ec21950d06f]) successfully deleted
W20260810 06:29:13.171447 986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:42901 (error 108)
I20260810 06:29:13.172348 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 1354
I20260810 06:29:13.172497 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.172166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:42901 (local address 127.0.0.1:46454)
0810 06:29:13.172337 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:13.172411 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:42901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 06:29:13.177413 986 catalog_manager.cc:4584] TS 1a1ccc890fc34347902bfca494ddbbf2 (127.4.73.1:42901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d847378e786c44cdbdd5a483e802573d: Network error: Client connection negotiation failed: client connection to 127.4.73.1:42901: connect: Connection refused (error 111)
I20260810 06:29:13.192245 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 1586
I20260810 06:29:13.205688 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 955
2026-08-10T06:29:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2553 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T06:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:13Z Disabled control of system clock
I20260810 06:29:13.245187 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38579
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:38579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:29:13.353403 1879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:13.353660 1879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:13.353693 1879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:13.357530 1879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:13.357580 1879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:13.357595 1879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:13.357613 1879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:13.357628 1879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:13.362308 1879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:38579
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:13.363227 1879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:13.364549 1879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:13.372725 1889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:13.372763 1892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:13.372807 1879 server_base.cc:1034] running on GCE node
W20260810 06:29:13.372725 1890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:13.373528 1879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:13.374459 1879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:13.375627 1879 hybrid_clock.cc:648] HybridClock initialized: now 1786343353375600 us; error 44 us; skew 500 ppm
I20260810 06:29:13.377722 1879 webserver.cc:460] Webserver started at http://127.4.73.62:42173/ using document root <none> and password file <none>
I20260810 06:29:13.378235 1879 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:13.378301 1879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:13.378460 1879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:13.380401 1879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "d2f677830fd24e71b686ba0332f37955"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "2bac5f17f4e8b9870cc93f144aa569a5"
server_key_iv: "2dcb618f16150bace736c9c1ae2ae8f2"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.380925 1879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "d2f677830fd24e71b686ba0332f37955"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "2bac5f17f4e8b9870cc93f144aa569a5"
server_key_iv: "2dcb618f16150bace736c9c1ae2ae8f2"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.384282 1879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 06:29:13.386523 1906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:13.387715 1879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 06:29:13.387868 1879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "d2f677830fd24e71b686ba0332f37955"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "2bac5f17f4e8b9870cc93f144aa569a5"
server_key_iv: "2dcb618f16150bace736c9c1ae2ae8f2"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.387993 1879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:13.419682 1879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:13.420553 1879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:13.420768 1879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:13.428045 1879 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:38579
I20260810 06:29:13.428057 2000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:38579 every 8 connection(s)
I20260810 06:29:13.429052 1879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:13.430950 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 1879
I20260810 06:29:13.431085 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:13.431428 4388 external_mini_cluster.cc:1428] Setting key 0186753ddec293ad26e3153e608f438f
I20260810 06:29:13.432760 2001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:13.438220 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955: Bootstrap starting.
I20260810 06:29:13.440449 2001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:13.440659 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.432876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.4.73.62:38579)
0810 06:29:13.433571 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:13.433593 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:13.433627 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:29:13.434237 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:13.434253 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:13.434643 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:13.434971 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:13.435384 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.436442 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.437362 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.437824 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.438020 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.438968 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:13.438973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:13.438984 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:13.439347 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:13.439402 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:13.439423 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:13.439532 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:13.439650 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 06:29:13.439894 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":144,"threads_started":1}
I20260810 06:29:13.441380 2001 log.cc:824] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:13.443320 2001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955: No bootstrap required, opened a new log
I20260810 06:29:13.446615 2001 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f677830fd24e71b686ba0332f37955" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38579 } }
I20260810 06:29:13.446823 2001 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:13.446858 2001 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f677830fd24e71b686ba0332f37955, State: Initialized, Role: FOLLOWER
I20260810 06:29:13.447248 2001 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f677830fd24e71b686ba0332f37955" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38579 } }
I20260810 06:29:13.447368 2001 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:13.447417 2001 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:13.447484 2001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:13.448271 2001 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f677830fd24e71b686ba0332f37955" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38579 } }
I20260810 06:29:13.448662 2001 leader_election.cc:302] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2f677830fd24e71b686ba0332f37955; no voters:
I20260810 06:29:13.449047 2001 leader_election.cc:288] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:13.449172 2009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:13.449420 2009 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [term 1 LEADER]: Becoming Leader. State: Replica: d2f677830fd24e71b686ba0332f37955, State: Running, Role: LEADER
I20260810 06:29:13.449702 2009 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f677830fd24e71b686ba0332f37955" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38579 } }
I20260810 06:29:13.450224 2001 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:13.451243 2015 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2f677830fd24e71b686ba0332f37955. Latest consensus state: current_term: 1 leader_uuid: "d2f677830fd24e71b686ba0332f37955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f677830fd24e71b686ba0332f37955" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38579 } } }
I20260810 06:29:13.451380 2015 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:13.451772 2027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:13.451740 2012 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2f677830fd24e71b686ba0332f37955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2f677830fd24e71b686ba0332f37955" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38579 } } }
I20260810 06:29:13.452425 2012 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:13.454300 2027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:13.459482 2027 catalog_manager.cc:1347] Generated new cluster ID: bc82c4a7efea4e69905fbb0b05708852
I20260810 06:29:13.459544 2027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:13.481354 2027 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:13.482259 2027 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:13.489516 2027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955: Generated new TSK 0
I20260810 06:29:13.490235 2027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:13.493763 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38579
--builtin_ntp_servers=127.4.73.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:29:13.605931 2047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:13.606173 2047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:13.606210 2047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:13.610105 2047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:13.610152 2047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:13.610203 2047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:29:13.614722 2047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:13.615587 2047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:13.616935 2047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:13.626405 2059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:13.626487 2047 server_base.cc:1034] running on GCE node
W20260810 06:29:13.626515 2056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:13.626407 2057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:13.627099 2047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:13.627697 2047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:13.628877 2047 hybrid_clock.cc:648] HybridClock initialized: now 1786343353628860 us; error 32 us; skew 500 ppm
I20260810 06:29:13.630585 2047 webserver.cc:460] Webserver started at http://127.4.73.1:40389/ using document root <none> and password file <none>
I20260810 06:29:13.631117 2047 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:13.631197 2047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:13.631449 2047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:13.633533 2047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "9bfb1fdb52b0463aba08189f6ed1f589"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "5d2c4d21c4469c3f09956a0f26ae0c3f"
server_key_iv: "94831cb5742fcc8592619269982724a9"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.634109 2047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "9bfb1fdb52b0463aba08189f6ed1f589"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "5d2c4d21c4469c3f09956a0f26ae0c3f"
server_key_iv: "94831cb5742fcc8592619269982724a9"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.637676 2047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:29:13.639979 2070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:13.641263 2047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:13.641392 2047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "9bfb1fdb52b0463aba08189f6ed1f589"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "5d2c4d21c4469c3f09956a0f26ae0c3f"
server_key_iv: "94831cb5742fcc8592619269982724a9"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.641506 2047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:13.661616 2047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:13.662344 2047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:13.662559 2047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:13.663157 2047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:13.664068 2047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:13.664140 2047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:13.664213 2047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:13.664258 2047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:13.673207 2047 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:36073
I20260810 06:29:13.673249 2267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:36073 every 8 connection(s)
I20260810 06:29:13.674059 2047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:29:13.681111 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 2047
I20260810 06:29:13.681218 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:29:13.681524 4388 external_mini_cluster.cc:1428] Setting key 7706670bee6cb61523bf40250c842615
I20260810 06:29:13.683004 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.675765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:38871 (local address 127.4.73.62:38579)
0810 06:29:13.675927 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:13.675931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:13.676995 (+ 1064us) server_negotiation.cc:407] Connection header received
0810 06:29:13.678328 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:13.678331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:13.678379 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:13.678451 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:13.679271 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.679782 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.680950 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.681185 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.681298 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.682463 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:13.682467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:13.682474 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:13.682487 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:13.682507 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:13.682515 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:13.682595 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:13.682760 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 06:29:13.682840 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 06:29:13.683593 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38579
--builtin_ntp_servers=127.4.73.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:29:13.683795 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.676174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38579 (local address 127.4.73.1:38871)
0810 06:29:13.676836 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:13.676890 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:13.678152 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:13.678613 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:13.678620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:13.678693 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:13.679148 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.679160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.679897 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.679899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.680803 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.680813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.681367 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.681374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.681531 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.682209 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:13.682211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:13.682249 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:13.682310 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:13.682597 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:13.682610 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:13.682617 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:29:13.682745 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 06:29:13.683013 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":163,"threads_started":1}
I20260810 06:29:13.685047 2269 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38579
I20260810 06:29:13.685264 2269 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:13.685863 2269 heartbeater.cc:511] Master 127.4.73.62:38579 requested a full tablet report, sending...
I20260810 06:29:13.687389 1939 ts_manager.cc:194] Registered new tserver with Master: 9bfb1fdb52b0463aba08189f6ed1f589 (127.4.73.1:36073)
I20260810 06:29:13.688766 1939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.1:38871
W20260810 06:29:13.794251 2276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:13.794502 2276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:13.794570 2276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:13.798372 2276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:13.798442 2276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:13.798524 2276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:29:13.803052 2276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:13.804044 2276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:13.805352 2276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:13.814814 2284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:13.814853 2276 server_base.cc:1034] running on GCE node
W20260810 06:29:13.815483 2290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:13.815493 2285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:13.815937 2276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:13.816524 2276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:13.817685 2276 hybrid_clock.cc:648] HybridClock initialized: now 1786343353817667 us; error 30 us; skew 500 ppm
I20260810 06:29:13.819424 2276 webserver.cc:460] Webserver started at http://127.4.73.2:41077/ using document root <none> and password file <none>
I20260810 06:29:13.819988 2276 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:13.820048 2276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:13.820292 2276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:13.822477 2276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "a8450d99521a4396bc3acba2a4297af6"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "c496cc917cf1a52183736488283cbff9"
server_key_iv: "564434e6cce6cd03cb575d8568ff9e8b"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.823048 2276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "a8450d99521a4396bc3acba2a4297af6"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "c496cc917cf1a52183736488283cbff9"
server_key_iv: "564434e6cce6cd03cb575d8568ff9e8b"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.826550 2276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 06:29:13.828907 2302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:13.830052 2276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:13.830183 2276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "a8450d99521a4396bc3acba2a4297af6"
format_stamp: "Formatted at 2026-08-10 06:29:13 on dist-test-slave-45dx"
server_key: "c496cc917cf1a52183736488283cbff9"
server_key_iv: "564434e6cce6cd03cb575d8568ff9e8b"
server_key_version: "encryptionkey@0"
I20260810 06:29:13.830315 2276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:13.866106 2276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:13.866863 2276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:13.867103 2276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:13.867735 2276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:13.868731 2276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:13.868778 2276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:13.868849 2276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:13.868891 2276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:13.878039 2276 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:46365
I20260810 06:29:13.878067 2506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:46365 every 8 connection(s)
I20260810 06:29:13.878897 2276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:29:13.881109 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 2276
I20260810 06:29:13.881263 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:29:13.881606 4388 external_mini_cluster.cc:1428] Setting key eebce6bb56db8f0ba9594ea2021695d3
I20260810 06:29:13.883798 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38579
--builtin_ntp_servers=127.4.73.20:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:29:13.888770 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.880679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:41887 (local address 127.4.73.62:38579)
0810 06:29:13.880842 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:13.880847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:13.882076 (+ 1229us) server_negotiation.cc:407] Connection header received
0810 06:29:13.883662 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:13.883665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:13.883708 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:13.883774 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:13.884985 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.885561 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.886724 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.887031 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.887115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.888226 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:13.888230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:13.888233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:13.888247 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:13.888273 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:13.888281 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:13.888409 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:13.888579 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 06:29:13.888638 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:29:13.889557 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:13.881100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38579 (local address 127.4.73.2:41887)
0810 06:29:13.881927 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:13.881982 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:13.883433 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:13.883959 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:13.883971 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:13.884078 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:13.884818 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.884841 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.885713 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.885716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.886604 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:13.886614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:13.887167 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:13.887170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:13.887306 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:13.887958 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:13.887960 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:13.887998 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:13.888058 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:13.888413 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:13.888424 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:13.888430 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 06:29:13.888579 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 06:29:13.888837 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":159,"threads_started":1}
I20260810 06:29:13.890641 2509 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38579
I20260810 06:29:13.890901 2509 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:13.891438 2509 heartbeater.cc:511] Master 127.4.73.62:38579 requested a full tablet report, sending...
I20260810 06:29:13.892489 1939 ts_manager.cc:194] Registered new tserver with Master: a8450d99521a4396bc3acba2a4297af6 (127.4.73.2:46365)
I20260810 06:29:13.893043 1939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.2:41887
W20260810 06:29:13.998041 2513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:13.998335 2513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:13.998416 2513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:14.002431 2513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:14.002516 2513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:14.002604 2513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:29:14.007418 2513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:14.008579 2513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:14.009930 2513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:14.019582 2520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:14.019615 2519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:14.019729 2513 server_base.cc:1034] running on GCE node
W20260810 06:29:14.019667 2524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:14.020273 2513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:14.020995 2513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:14.022159 2513 hybrid_clock.cc:648] HybridClock initialized: now 1786343354022128 us; error 43 us; skew 500 ppm
I20260810 06:29:14.024380 2513 webserver.cc:460] Webserver started at http://127.4.73.3:44855/ using document root <none> and password file <none>
I20260810 06:29:14.024996 2513 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:14.025063 2513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:14.025311 2513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:14.027418 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "fff88baaec244d69995a5d8704fe1e37"
format_stamp: "Formatted at 2026-08-10 06:29:14 on dist-test-slave-45dx"
server_key: "4e8a896f3acbce45a83e5686fae01ceb"
server_key_iv: "dcebab1262b885f2dd053b081fc8345f"
server_key_version: "encryptionkey@0"
I20260810 06:29:14.028030 2513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "fff88baaec244d69995a5d8704fe1e37"
format_stamp: "Formatted at 2026-08-10 06:29:14 on dist-test-slave-45dx"
server_key: "4e8a896f3acbce45a83e5686fae01ceb"
server_key_iv: "dcebab1262b885f2dd053b081fc8345f"
server_key_version: "encryptionkey@0"
I20260810 06:29:14.031919 2513 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 06:29:14.034569 2534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:14.035853 2513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:14.035992 2513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "fff88baaec244d69995a5d8704fe1e37"
format_stamp: "Formatted at 2026-08-10 06:29:14 on dist-test-slave-45dx"
server_key: "4e8a896f3acbce45a83e5686fae01ceb"
server_key_iv: "dcebab1262b885f2dd053b081fc8345f"
server_key_version: "encryptionkey@0"
I20260810 06:29:14.036100 2513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:14.061652 2513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:14.062374 2513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:14.062587 2513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:14.063211 2513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:14.064163 2513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:14.064213 2513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:14.064281 2513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:14.064329 2513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:14.072956 2513 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:43077
I20260810 06:29:14.072989 2739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:43077 every 8 connection(s)
I20260810 06:29:14.073884 2513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:29:14.081661 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 2513
I20260810 06:29:14.081766 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:29:14.082157 4388 external_mini_cluster.cc:1428] Setting key 64a0a34510e1e46f82147cacd0ca36c1
I20260810 06:29:14.082887 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.075678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:46529 (local address 127.4.73.62:38579)
0810 06:29:14.075829 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.075834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.076954 (+ 1120us) server_negotiation.cc:407] Connection header received
0810 06:29:14.078287 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.078290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.078335 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.078414 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:14.079250 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.079737 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.080823 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.081037 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.081115 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.082396 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:14.082399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:14.082402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:14.082414 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:14.082433 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:14.082441 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:14.082509 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.082698 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.082769 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:29:14.083993 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.076112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38579 (local address 127.4.73.3:46529)
0810 06:29:14.076801 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:14.076847 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:14.078107 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:14.078597 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:14.078602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:14.078679 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:14.079132 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.079143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.079838 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.079840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.080720 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.080730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.081169 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.081172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.081309 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.082044 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:14.082046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:14.082104 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:14.082186 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:14.082544 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:14.082557 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:14.082562 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 06:29:14.082667 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:29:14.082973 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":167,"threads_started":1}
I20260810 06:29:14.085160 2743 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38579
I20260810 06:29:14.085393 2743 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:14.085880 2743 heartbeater.cc:511] Master 127.4.73.62:38579 requested a full tablet report, sending...
I20260810 06:29:14.086814 1939 ts_manager.cc:194] Registered new tserver with Master: fff88baaec244d69995a5d8704fe1e37 (127.4.73.3:43077)
I20260810 06:29:14.087410 1939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.3:46529
I20260810 06:29:14.096382 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:29:14.102178 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.098283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52410 (local address 127.4.73.62:38579)
0810 06:29:14.098434 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.098440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.098606 (+ 166us) server_negotiation.cc:407] Connection header received
0810 06:29:14.098771 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.098774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.098813 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.098917 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:14.099189 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.099676 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.100597 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.100809 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.100925 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.101758 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:14.101761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:14.101764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:14.101775 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:14.101794 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:14.101802 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:14.101888 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.101981 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.102035 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 06:29:14.104979 1939 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:29:14.107918 1939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:29:14.122048 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.115949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43077 (local address 127.0.0.1:36004)
0810 06:29:14.116643 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:14.116658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:14.116743 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:14.117623 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:14.117627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:14.117642 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:14.117780 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.117788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.119043 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.119050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.120569 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.120581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.121478 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.121482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.121591 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.121651 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 06:29:14.121771 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 06:29:14.121888 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":76,"thread_start_us":129,"threads_started":1}
I20260810 06:29:14.122238 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.117205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.4.73.1:36073)
0810 06:29:14.117709 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.117722 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.117745 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:29:14.117829 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.117833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.118043 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.118208 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:14.118436 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.119470 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.120976 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.121568 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.121678 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.121879 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.121968 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.122064 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":92,"threads_started":1}
I20260810 06:29:14.122200 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.116474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:36073 (local address 127.0.0.1:44274)
0810 06:29:14.117244 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:14.117257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:14.117348 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:14.118201 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:14.118204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:14.118222 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:14.118317 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.118323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.119582 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.119587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.120837 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.120848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.121704 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.121707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.121821 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.121846 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:29:14.121930 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 06:29:14.121995 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":87,"threads_started":1}
I20260810 06:29:14.123016 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.116757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.4.73.3:43077)
0810 06:29:14.117143 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.117161 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.117184 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:29:14.117258 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.117266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.117469 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.117944 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:14.118061 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.118928 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.120705 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.121340 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.121435 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.121574 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.122764 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.122842 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":137,"threads_started":1}
I20260810 06:29:14.123239 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.115780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:46365 (local address 127.0.0.1:53880)
0810 06:29:14.116328 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:14.116379 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:14.116515 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:14.117498 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:14.117504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:14.117530 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:14.117796 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.117802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.119227 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.119230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.120569 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.120581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.122846 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.122849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.122961 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.122983 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:29:14.123063 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:29:14.123123 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":91,"thread_start_us":60,"threads_started":1}
I20260810 06:29:14.123663 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.115946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.4.73.2:46365)
0810 06:29:14.116652 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.116664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.116685 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:29:14.116756 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.116759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.116949 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.117659 (+ 710us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:14.117913 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.119108 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.121653 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.122478 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.122586 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.122779 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.123426 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.123515 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":137,"threads_started":1}
I20260810 06:29:14.125553 2158 tablet_service.cc:1467] Processing CreateTablet for tablet dc359bd1d1784aa59f8f0acfdfb194f4 (DEFAULT_TABLE table=test-table [id=8bfd6e28dcf9462a820b89d0390d74f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:14.126091 2620 tablet_service.cc:1467] Processing CreateTablet for tablet dc359bd1d1784aa59f8f0acfdfb194f4 (DEFAULT_TABLE table=test-table [id=8bfd6e28dcf9462a820b89d0390d74f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:14.126709 2158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc359bd1d1784aa59f8f0acfdfb194f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:14.127091 2620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc359bd1d1784aa59f8f0acfdfb194f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:14.127902 2390 tablet_service.cc:1467] Processing CreateTablet for tablet dc359bd1d1784aa59f8f0acfdfb194f4 (DEFAULT_TABLE table=test-table [id=8bfd6e28dcf9462a820b89d0390d74f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:14.128695 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc359bd1d1784aa59f8f0acfdfb194f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:14.134657 2775 tablet_bootstrap.cc:492] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589: Bootstrap starting.
I20260810 06:29:14.136153 2776 tablet_bootstrap.cc:492] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Bootstrap starting.
I20260810 06:29:14.136365 2774 tablet_bootstrap.cc:492] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Bootstrap starting.
I20260810 06:29:14.138538 2774 tablet_bootstrap.cc:654] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:14.138538 2776 tablet_bootstrap.cc:654] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:14.139482 2774 log.cc:824] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:14.139478 2776 log.cc:824] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:14.140549 2775 tablet_bootstrap.cc:654] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:14.141409 2775 log.cc:824] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:14.141855 2774 tablet_bootstrap.cc:492] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: No bootstrap required, opened a new log
I20260810 06:29:14.142055 2776 tablet_bootstrap.cc:492] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: No bootstrap required, opened a new log
I20260810 06:29:14.142132 2774 ts_tablet_manager.cc:1397] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 06:29:14.142254 2776 ts_tablet_manager.cc:1397] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:29:14.143419 2775 tablet_bootstrap.cc:492] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589: No bootstrap required, opened a new log
I20260810 06:29:14.143596 2775 ts_tablet_manager.cc:1397] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260810 06:29:14.146590 2774 raft_consensus.cc:348] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.146857 2774 raft_consensus.cc:374] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:14.146724 2776 raft_consensus.cc:348] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.146942 2774 raft_consensus.cc:729] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff88baaec244d69995a5d8704fe1e37, State: Initialized, Role: FOLLOWER
I20260810 06:29:14.147023 2776 raft_consensus.cc:374] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:14.147092 2776 raft_consensus.cc:729] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8450d99521a4396bc3acba2a4297af6, State: Initialized, Role: FOLLOWER
I20260810 06:29:14.147527 2774 consensus_queue.cc:260] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.147540 2776 consensus_queue.cc:260] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.147890 2775 raft_consensus.cc:348] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.148207 2775 raft_consensus.cc:374] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:14.148272 2775 raft_consensus.cc:729] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfb1fdb52b0463aba08189f6ed1f589, State: Initialized, Role: FOLLOWER
I20260810 06:29:14.148254 2509 heartbeater.cc:503] Master 127.4.73.62:38579 was elected leader, sending a full tablet report...
I20260810 06:29:14.148622 2776 ts_tablet_manager.cc:1428] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 06:29:14.148663 2775 consensus_queue.cc:260] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.149171 2743 heartbeater.cc:503] Master 127.4.73.62:38579 was elected leader, sending a full tablet report...
I20260810 06:29:14.149545 2775 ts_tablet_manager.cc:1428] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:29:14.149891 2774 ts_tablet_manager.cc:1428] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 06:29:14.150095 2269 heartbeater.cc:503] Master 127.4.73.62:38579 was elected leader, sending a full tablet report...
W20260810 06:29:14.175283 2271 tablet.cc:2369] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:29:14.325119 2746 tablet.cc:2369] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:29:14.380146 2510 tablet.cc:2369] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:14.388550 2781 raft_consensus.cc:482] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:14.388753 2781 raft_consensus.cc:504] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.389950 2781 leader_election.cc:288] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fff88baaec244d69995a5d8704fe1e37 (127.4.73.3:43077), 9bfb1fdb52b0463aba08189f6ed1f589 (127.4.73.1:36073)
I20260810 06:29:14.394285 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.390518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:50709 (local address 127.4.73.3:43077)
0810 06:29:14.390685 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.390692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.390714 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:29:14.390780 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.390783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.390875 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.390975 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:14.391299 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.392035 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.393160 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.393695 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.393779 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.393829 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.394064 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.394148 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:29:14.394280 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.390256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:43077 (local address 127.4.73.2:50709)
0810 06:29:14.390510 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:14.390527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:14.390633 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:14.391000 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:14.391004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:14.391029 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:14.391172 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.391181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.392147 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.392150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.393039 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.393049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.393818 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.393821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.393934 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.393957 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:29:14.394040 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 06:29:14.394097 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"spinlock_wait_cycles":22272,"thread_start_us":99,"threads_started":1}
I20260810 06:29:14.394430 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.390972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:51185 (local address 127.4.73.1:36073)
0810 06:29:14.391094 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.391098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.391112 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:29:14.391220 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.391226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.391286 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.391362 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:14.391665 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.392344 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.393400 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.393821 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.393925 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.393984 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.394212 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.394292 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:29:14.394452 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.390683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:36073 (local address 127.4.73.2:51185)
0810 06:29:14.390984 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:14.391001 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:14.391102 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:14.391414 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:14.391417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:14.391435 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:14.391554 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.391560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.392464 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.392466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.393253 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:14.393263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.393965 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.393968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:14.394077 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.394101 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:29:14.394179 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:29:14.394245 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260810 06:29:14.395117 2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc359bd1d1784aa59f8f0acfdfb194f4" candidate_uuid: "a8450d99521a4396bc3acba2a4297af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" is_pre_election: true
I20260810 06:29:14.395120 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc359bd1d1784aa59f8f0acfdfb194f4" candidate_uuid: "a8450d99521a4396bc3acba2a4297af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff88baaec244d69995a5d8704fe1e37" is_pre_election: true
I20260810 06:29:14.395444 2659 raft_consensus.cc:2454] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8450d99521a4396bc3acba2a4297af6 in term 0.
I20260810 06:29:14.395444 2194 raft_consensus.cc:2454] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8450d99521a4396bc3acba2a4297af6 in term 0.
I20260810 06:29:14.395953 2304 leader_election.cc:302] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8450d99521a4396bc3acba2a4297af6, fff88baaec244d69995a5d8704fe1e37; no voters:
I20260810 06:29:14.396190 2781 raft_consensus.cc:2781] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:29:14.396368 2781 raft_consensus.cc:482] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:29:14.396426 2781 raft_consensus.cc:3037] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:14.397619 2781 raft_consensus.cc:504] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.397935 2781 leader_election.cc:288] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [CANDIDATE]: Term 1 election: Requested vote from peers fff88baaec244d69995a5d8704fe1e37 (127.4.73.3:43077), 9bfb1fdb52b0463aba08189f6ed1f589 (127.4.73.1:36073)
I20260810 06:29:14.398398 2659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc359bd1d1784aa59f8f0acfdfb194f4" candidate_uuid: "a8450d99521a4396bc3acba2a4297af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff88baaec244d69995a5d8704fe1e37"
I20260810 06:29:14.398427 2194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc359bd1d1784aa59f8f0acfdfb194f4" candidate_uuid: "a8450d99521a4396bc3acba2a4297af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfb1fdb52b0463aba08189f6ed1f589"
I20260810 06:29:14.398543 2194 raft_consensus.cc:3037] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:14.398542 2659 raft_consensus.cc:3037] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:14.399827 2194 raft_consensus.cc:2454] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8450d99521a4396bc3acba2a4297af6 in term 1.
I20260810 06:29:14.399827 2659 raft_consensus.cc:2454] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8450d99521a4396bc3acba2a4297af6 in term 1.
I20260810 06:29:14.400137 2304 leader_election.cc:302] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8450d99521a4396bc3acba2a4297af6, fff88baaec244d69995a5d8704fe1e37; no voters:
I20260810 06:29:14.400321 2781 raft_consensus.cc:2781] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:14.400552 2781 raft_consensus.cc:686] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 1 LEADER]: Becoming Leader. State: Replica: a8450d99521a4396bc3acba2a4297af6, State: Running, Role: LEADER
I20260810 06:29:14.400879 2781 consensus_queue.cc:237] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } }
I20260810 06:29:14.404170 1937 catalog_manager.cc:5495] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8450d99521a4396bc3acba2a4297af6 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "a8450d99521a4396bc3acba2a4297af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8450d99521a4396bc3acba2a4297af6" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 } health_report { overall_health: UNKNOWN } } }
I20260810 06:29:14.483690 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.480169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.4.73.2:46365)
0810 06:29:14.480348 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:14.480352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:14.480365 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:14.480500 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:14.480502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:14.480564 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:14.480659 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:14.480935 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.481441 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.482345 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:14.482552 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:14.482628 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:14.482906 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:14.483319 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:14.483409 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:14.483535 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 06:29:14.483581 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 06:29:14.489493 2659 raft_consensus.cc:1261] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 1 FOLLOWER]: Refusing update from remote peer a8450d99521a4396bc3acba2a4297af6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:14.489493 2194 raft_consensus.cc:1261] T dc359bd1d1784aa59f8f0acfdfb194f4 P 9bfb1fdb52b0463aba08189f6ed1f589 [term 1 FOLLOWER]: Refusing update from remote peer a8450d99521a4396bc3acba2a4297af6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:14.490204 2790 consensus_queue.cc:1035] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fff88baaec244d69995a5d8704fe1e37" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 43077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:14.490384 2781 consensus_queue.cc:1035] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfb1fdb52b0463aba08189f6ed1f589" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 36073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:14.496989 2794 mvcc.cc:205] Tried to move back new op lower bound from 7316862379981475840 to 7316862379628605440. Current Snapshot: MvccSnapshot[applied={T|T < 7316862379981475840}]
I20260810 06:29:14.501008 2797 mvcc.cc:205] Tried to move back new op lower bound from 7316862379981475840 to 7316862379628605440. Current Snapshot: MvccSnapshot[applied={T|T < 7316862379981475840}]
I20260810 06:29:14.505046 1939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:14.505247 1939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:14.507655 1939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2f677830fd24e71b686ba0332f37955: Sending DeleteTablet for 3 replicas of tablet dc359bd1d1784aa59f8f0acfdfb194f4
I20260810 06:29:14.508494 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet dc359bd1d1784aa59f8f0acfdfb194f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:14 UTC) from {username='slave'} at 127.0.0.1:53880
I20260810 06:29:14.508525 2620 tablet_service.cc:1514] Processing DeleteTablet for tablet dc359bd1d1784aa59f8f0acfdfb194f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:14 UTC) from {username='slave'} at 127.0.0.1:36004
I20260810 06:29:14.508519 2158 tablet_service.cc:1514] Processing DeleteTablet for tablet dc359bd1d1784aa59f8f0acfdfb194f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:14 UTC) from {username='slave'} at 127.0.0.1:44274
I20260810 06:29:14.509011 2808 tablet_replica.cc:330] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: stopping tablet replica
I20260810 06:29:14.509367 2808 raft_consensus.cc:2229] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:14.509671 2808 raft_consensus.cc:2258] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:14.509727 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 2047
I20260810 06:29:14.533370 2810 tablet_replica.cc:330] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: stopping tablet replica
I20260810 06:29:14.533797 2810 raft_consensus.cc:2229] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:29:14.534278 2810 raft_consensus.cc:2258] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:14.536079 2808 ts_tablet_manager.cc:1905] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:14.537482 2810 ts_tablet_manager.cc:1905] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:14.539773 2808 ts_tablet_manager.cc:1918] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:14.539950 2808 log.cc:1196] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/dc359bd1d1784aa59f8f0acfdfb194f4
I20260810 06:29:14.540273 2810 ts_tablet_manager.cc:1918] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:14.540395 2808 ts_tablet_manager.cc:1939] T dc359bd1d1784aa59f8f0acfdfb194f4 P fff88baaec244d69995a5d8704fe1e37: Deleting consensus metadata
I20260810 06:29:14.540493 2810 log.cc:1196] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/dc359bd1d1784aa59f8f0acfdfb194f4
I20260810 06:29:14.540923 2810 ts_tablet_manager.cc:1939] T dc359bd1d1784aa59f8f0acfdfb194f4 P a8450d99521a4396bc3acba2a4297af6: Deleting consensus metadata
I20260810 06:29:14.542052 1915 catalog_manager.cc:4841] TS a8450d99521a4396bc3acba2a4297af6 (127.4.73.2:46365): tablet dc359bd1d1784aa59f8f0acfdfb194f4 (table test-table [id=8bfd6e28dcf9462a820b89d0390d74f2]) successfully deleted
I20260810 06:29:14.542374 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 2276
W20260810 06:29:14.542402 1913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:36073 (error 108)
I20260810 06:29:14.542950 1913 catalog_manager.cc:4841] TS fff88baaec244d69995a5d8704fe1e37 (127.4.73.3:43077): tablet dc359bd1d1784aa59f8f0acfdfb194f4 (table test-table [id=8bfd6e28dcf9462a820b89d0390d74f2]) successfully deleted
I20260810 06:29:14.548707 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:14.548267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:36073 (local address 127.0.0.1:44282)
0810 06:29:14.548517 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:14.548600 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:36073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260810 06:29:14.548923 1913 catalog_manager.cc:4584] TS 9bfb1fdb52b0463aba08189f6ed1f589 (127.4.73.1:36073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc359bd1d1784aa59f8f0acfdfb194f4: Network error: Client connection negotiation failed: client connection to 127.4.73.1:36073: connect: Connection refused (error 111)
I20260810 06:29:14.555604 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 2513
I20260810 06:29:14.569288 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 1879
2026-08-10T06:29:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1359 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:14 dist-test-slave-45dx krb5kdc[2816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:14 dist-test-slave-45dx krb5kdc[2816](info): set up 2 sockets
Aug 10 06:29:14 dist-test-slave-45dx krb5kdc[2816](info): commencing operation
krb5kdc: starting...
W20260810 06:29:15.813670 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:15 dist-test-slave-45dx krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:15.930632 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:39063
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:16.035596 2841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:16.035879 2841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:16.035918 2841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:16.039857 2841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:16.039911 2841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:16.039930 2841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:16.039947 2841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:16.039960 2841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:16.045014 2841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:16.045924 2841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:16.046825 2841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:16.055100 2852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:16.055122 2854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:16.055224 2841 server_base.cc:1034] running on GCE node
W20260810 06:29:16.055346 2849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:16.055974 2841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:16.057075 2841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:16.058241 2841 hybrid_clock.cc:648] HybridClock initialized: now 1786343356058206 us; error 48 us; skew 500 ppm
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:16.060993 2841 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:16.061774 2841 webserver.cc:460] Webserver started at http://127.4.73.62:35383/ using document root <none> and password file <none>
I20260810 06:29:16.062250 2841 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:16.062305 2841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:16.062469 2841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:16.064512 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "454f578e2e734f078afe95ca89342188"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "4a8a3e9ab0f576394d7ca12552b613a7"
server_key_iv: "124d7f830846d0cd8def236fc26971bf"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.065011 2841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "454f578e2e734f078afe95ca89342188"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "4a8a3e9ab0f576394d7ca12552b613a7"
server_key_iv: "124d7f830846d0cd8def236fc26971bf"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.068583 2841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:29:16.070930 2866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.072100 2841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:16.072252 2841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "454f578e2e734f078afe95ca89342188"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "4a8a3e9ab0f576394d7ca12552b613a7"
server_key_iv: "124d7f830846d0cd8def236fc26971bf"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.072433 2841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:16.083498 2841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:16.086148 2841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:16.086334 2841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:16.092862 2841 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:39063
I20260810 06:29:16.092924 2960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:39063 every 8 connection(s)
I20260810 06:29:16.093767 2841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:16.095887 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 2841
I20260810 06:29:16.096072 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:16.096436 4388 external_mini_cluster.cc:1428] Setting key 60a014b09adf5c1367568b0f789c398d
I20260810 06:29:16.096446 2961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:16.103421 2961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188: Bootstrap starting.
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:16.105563 2961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:16.106395 2961 log.cc:824] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:16.108139 2961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188: No bootstrap required, opened a new log
I20260810 06:29:16.110555 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.097686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.4.73.62:39063)
0810 06:29:16.098096 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.098113 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.098152 (+ 39us) server_negotiation.cc:407] Connection header received
0810 06:29:16.099160 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.099182 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.099559 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.099881 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:16.100560 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.101994 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.102848 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.103524 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.103687 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.106078 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:16.106084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:16.106097 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:16.106126 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:16.107876 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.108379 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.108384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.108388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.108495 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.108781 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.108787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.108791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.109199 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:16.109414 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.109643 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.109934 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260810 06:29:16.111891 2961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454f578e2e734f078afe95ca89342188" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39063 } }
I20260810 06:29:16.112130 2961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:16.112210 2961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 454f578e2e734f078afe95ca89342188, State: Initialized, Role: FOLLOWER
I20260810 06:29:16.112696 2961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454f578e2e734f078afe95ca89342188" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39063 } }
I20260810 06:29:16.112874 2961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:16.112969 2961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:16.113065 2961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:16.113844 2961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454f578e2e734f078afe95ca89342188" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39063 } }
I20260810 06:29:16.114167 2961 leader_election.cc:302] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 454f578e2e734f078afe95ca89342188; no voters:
I20260810 06:29:16.114459 2961 leader_election.cc:288] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:16.114598 2967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:16.114820 2967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [term 1 LEADER]: Becoming Leader. State: Replica: 454f578e2e734f078afe95ca89342188, State: Running, Role: LEADER
I20260810 06:29:16.115157 2967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454f578e2e734f078afe95ca89342188" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39063 } }
I20260810 06:29:16.115646 2961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:16.116752 2972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 454f578e2e734f078afe95ca89342188. Latest consensus state: current_term: 1 leader_uuid: "454f578e2e734f078afe95ca89342188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454f578e2e734f078afe95ca89342188" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39063 } } }
I20260810 06:29:16.116915 2972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:16.117228 2971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "454f578e2e734f078afe95ca89342188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "454f578e2e734f078afe95ca89342188" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39063 } } }
I20260810 06:29:16.117327 2984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:16.120528 2984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:16.122081 2971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:16.126070 2984 catalog_manager.cc:1347] Generated new cluster ID: 3dbe1bb288664080adaa2051c56c3268
I20260810 06:29:16.126138 2984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:16.143654 2984 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:16.144974 2984 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:16.150218 2984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188: Generated new TSK 0
I20260810 06:29:16.150774 2984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:29:16.182618 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39063
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:16.307713 3011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:16.307950 3011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:16.307991 3011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:16.311753 3011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:16.311802 3011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:16.311856 3011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:29:16.316624 3011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:16.317596 3011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:16.318598 3011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:16.328068 3024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:16.328059 3023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:16.328056 3026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:16.328477 3011 server_base.cc:1034] running on GCE node
I20260810 06:29:16.328892 3011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:16.329486 3011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:16.330678 3011 hybrid_clock.cc:648] HybridClock initialized: now 1786343356330660 us; error 28 us; skew 500 ppm
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:16.333469 3011 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:29:16.334250 3011 webserver.cc:460] Webserver started at http://127.4.73.1:44955/ using document root <none> and password file <none>
I20260810 06:29:16.334779 3011 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:16.334838 3011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:16.335047 3011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:16.337024 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "8d511ac5936a4409916fd49cf7d6f109"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "6cbc23a4da969d1e7eea7e1f9f045dbc"
server_key_iv: "ae8dcb66c262596e3dbdc6fb63f18480"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.337600 3011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "8d511ac5936a4409916fd49cf7d6f109"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "6cbc23a4da969d1e7eea7e1f9f045dbc"
server_key_iv: "ae8dcb66c262596e3dbdc6fb63f18480"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.341141 3011 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:16.343434 3038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.344651 3011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:16.344767 3011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "8d511ac5936a4409916fd49cf7d6f109"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "6cbc23a4da969d1e7eea7e1f9f045dbc"
server_key_iv: "ae8dcb66c262596e3dbdc6fb63f18480"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.344867 3011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:16.360574 3011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:16.363659 3011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:16.363867 3011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:16.364454 3011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:16.365207 3011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:16.365254 3011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.365321 3011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:16.365353 3011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.373730 3011 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:33133
I20260810 06:29:16.373761 3244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:33133 every 8 connection(s)
I20260810 06:29:16.374573 3011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:29:16.379241 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 3011
I20260810 06:29:16.379376 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:29:16.379664 4388 external_mini_cluster.cc:1428] Setting key 4696098ef0bcb73454c05435b52e7796
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:16.389096 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.376233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:38721 (local address 127.4.73.62:39063)
0810 06:29:16.376407 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.376411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.377213 (+ 802us) server_negotiation.cc:407] Connection header received
0810 06:29:16.378551 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.378554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.378612 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.378708 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:16.379924 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.380599 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.381874 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.382116 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.382228 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.384967 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:16.384974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:16.384980 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:16.385015 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:16.386891 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.387393 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.387397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.387400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.387506 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.387845 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.387858 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.387862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.388127 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:16.388254 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.388744 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.388895 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:29:16.389635 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.376641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39063 (local address 127.4.73.1:38721)
0810 06:29:16.377052 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:16.377109 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:16.378359 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:16.378864 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:16.378870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:16.379209 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:16.379777 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.379792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.380740 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.380743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.381711 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.381721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.382263 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.382268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.382433 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.383043 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:16.383066 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:16.384804 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:16.387015 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:16.387023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:16.387033 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:16.387287 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:16.387614 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:16.387616 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:16.387618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:16.387745 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:16.388248 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:16.388252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:16.388514 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 06:29:16.388659 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 06:29:16.388881 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
I20260810 06:29:16.390786 3245 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39063
I20260810 06:29:16.391057 3245 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:16.391464 3245 heartbeater.cc:511] Master 127.4.73.62:39063 requested a full tablet report, sending...
I20260810 06:29:16.392984 2902 ts_manager.cc:194] Registered new tserver with Master: 8d511ac5936a4409916fd49cf7d6f109 (127.4.73.1:33133)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:29:16.394598 2902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:38721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:29:16.408542 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39063
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:16.515641 3256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:16.515904 3256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:16.515986 3256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:16.519977 3256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:16.520048 3256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:16.520135 3256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:29:16.525012 3256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:16.525887 3256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:16.526783 3256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:16.536195 3266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:16.536233 3256 server_base.cc:1034] running on GCE node
W20260810 06:29:16.536249 3265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:16.536229 3269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:16.536892 3256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:16.537470 3256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:16.538663 3256 hybrid_clock.cc:648] HybridClock initialized: now 1786343356538652 us; error 35 us; skew 500 ppm
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:16.541303 3256 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:29:16.542179 3256 webserver.cc:460] Webserver started at http://127.4.73.2:43561/ using document root <none> and password file <none>
I20260810 06:29:16.542711 3256 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:16.542802 3256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:16.543028 3256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:16.545042 3256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "76c1a0a4dd9e460082582e64747a363a"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "13fc622e184c5dc245845f20e2dae20f"
server_key_iv: "f6d78671c4238b4d26330e9bfee46e16"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.545591 3256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "76c1a0a4dd9e460082582e64747a363a"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "13fc622e184c5dc245845f20e2dae20f"
server_key_iv: "f6d78671c4238b4d26330e9bfee46e16"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.549114 3256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:16.551332 3280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.552536 3256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:16.552661 3256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "76c1a0a4dd9e460082582e64747a363a"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "13fc622e184c5dc245845f20e2dae20f"
server_key_iv: "f6d78671c4238b4d26330e9bfee46e16"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.552765 3256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:16.563242 3256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:16.565914 3256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:16.566097 3256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:16.566608 3256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:16.567361 3256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:16.567435 3256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.567516 3256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:16.567572 3256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.576545 3256 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:37167
I20260810 06:29:16.576578 3475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:37167 every 8 connection(s)
I20260810 06:29:16.577389 3256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:29:16.585130 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 3256
I20260810 06:29:16.585274 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:29:16.585579 4388 external_mini_cluster.cc:1428] Setting key 39d64804326677e86fae750ac8f0c825
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:16.590819 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.578997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:56167 (local address 127.4.73.62:39063)
0810 06:29:16.579136 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.579140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.579866 (+ 726us) server_negotiation.cc:407] Connection header received
0810 06:29:16.581214 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.581217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.581276 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.581350 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:16.582432 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.582972 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.584060 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.584296 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.584452 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.587039 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:16.587045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:16.587050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:16.587077 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:16.588602 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.589227 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.589232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.589235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.589336 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.589658 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.589663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.589666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.589897 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:16.590011 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.590464 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.590607 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:29:16.591391 3479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.579292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39063 (local address 127.4.73.2:56167)
0810 06:29:16.579712 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:16.579749 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:16.581019 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:16.581506 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:16.581512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:16.581861 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:16.582286 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.582299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.583113 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.583121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.583936 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.583945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.584503 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.584507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.584657 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.585151 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:16.585173 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:16.586847 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:16.588713 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:16.588721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:16.588737 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:16.589105 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:16.589432 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:16.589434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:16.589435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:16.589564 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:16.590003 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:16.590007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:16.590226 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 06:29:16.590381 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 06:29:16.590615 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":117,"threads_started":1}
I20260810 06:29:16.592546 3476 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39063
I20260810 06:29:16.592787 3476 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:16.593321 3476 heartbeater.cc:511] Master 127.4.73.62:39063 requested a full tablet report, sending...
I20260810 06:29:16.594424 2902 ts_manager.cc:194] Registered new tserver with Master: 76c1a0a4dd9e460082582e64747a363a (127.4.73.2:37167)
I20260810 06:29:16.594992 2902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:56167
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:29:16.613325 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39063
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:16.718338 3484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:16.718616 3484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:16.718693 3484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:16.722543 3484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:16.722622 3484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:16.722707 3484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:29:16.727464 3484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:39063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:16.728514 3484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:16.729468 3484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:16.738780 3496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:16.738854 3492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:16.738781 3491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:16.738993 3484 server_base.cc:1034] running on GCE node
I20260810 06:29:16.739526 3484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:16.740103 3484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:16.741282 3484 hybrid_clock.cc:648] HybridClock initialized: now 1786343356741255 us; error 38 us; skew 500 ppm
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:16.743906 3484 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:29:16.744714 3484 webserver.cc:460] Webserver started at http://127.4.73.3:43343/ using document root <none> and password file <none>
I20260810 06:29:16.745222 3484 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:16.745311 3484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:16.745546 3484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:16.747488 3484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "abd9e19595194560a9167cc5422ce5f5"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "6845b32fcbce61bd4534e7691e0b91f8"
server_key_iv: "f6666861df0716423e09c250d5212d40"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.748083 3484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "abd9e19595194560a9167cc5422ce5f5"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "6845b32fcbce61bd4534e7691e0b91f8"
server_key_iv: "f6666861df0716423e09c250d5212d40"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.751550 3484 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:29:16.753762 3509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.754881 3484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:29:16.755033 3484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "abd9e19595194560a9167cc5422ce5f5"
format_stamp: "Formatted at 2026-08-10 06:29:16 on dist-test-slave-45dx"
server_key: "6845b32fcbce61bd4534e7691e0b91f8"
server_key_iv: "f6666861df0716423e09c250d5212d40"
server_key_version: "encryptionkey@0"
I20260810 06:29:16.755136 3484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:16.787581 3484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:16.790726 3484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:16.790947 3484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:16.791545 3484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:16.792389 3484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:16.792464 3484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.792537 3484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:16.792589 3484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:16.801213 3484 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:41557
I20260810 06:29:16.801239 3710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:41557 every 8 connection(s)
I20260810 06:29:16.802079 3484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:29:16.809429 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 3484
I20260810 06:29:16.809562 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:29:16.809880 4388 external_mini_cluster.cc:1428] Setting key 426f9905e1e44b976f1ecd433421bbd2
Aug 10 06:29:16 dist-test-slave-45dx krb5kdc[2816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343356, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:16.815259 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.803887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:38973 (local address 127.4.73.62:39063)
0810 06:29:16.804020 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.804025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.804792 (+ 767us) server_negotiation.cc:407] Connection header received
0810 06:29:16.806119 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.806122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.806181 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.806289 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:16.807428 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.807954 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.809189 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.809534 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.809646 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.812202 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:16.812205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:16.812207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:16.812231 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:16.813526 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.814013 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.814015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.814017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.814092 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.814383 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.814385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.814386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.814547 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:16.814623 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.815043 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.815136 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:29:16.816099 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.804212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:39063 (local address 127.4.73.3:38973)
0810 06:29:16.804636 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:16.804675 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:16.805917 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:16.806463 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:16.806469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:16.806804 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:16.807274 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.807287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.808084 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.808093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.809053 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.809063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.809697 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.809701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.809843 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.810542 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:16.810558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:16.812060 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:16.813649 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:16.813653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:16.813663 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:16.813914 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:16.814171 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:16.814173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:16.814174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:16.814298 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:16.814633 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:16.814637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:16.814842 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 06:29:16.814996 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 06:29:16.815211 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
I20260810 06:29:16.817230 3711 heartbeater.cc:347] Connected to a master server at 127.4.73.62:39063
I20260810 06:29:16.817435 3711 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:16.817836 3711 heartbeater.cc:511] Master 127.4.73.62:39063 requested a full tablet report, sending...
I20260810 06:29:16.818773 2902 ts_manager.cc:194] Registered new tserver with Master: abd9e19595194560a9167cc5422ce5f5 (127.4.73.3:41557)
I20260810 06:29:16.819317 2902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:38973
I20260810 06:29:16.823577 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:29:16.833063 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.825513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.4.73.62:39063)
0810 06:29:16.825654 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.825657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.825913 (+ 256us) server_negotiation.cc:407] Connection header received
0810 06:29:16.826006 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.826008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.826042 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.826105 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:16.826808 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.827288 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.828134 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.828444 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.828523 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.829943 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:16.829945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:16.829947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:16.829968 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:16.831385 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.831870 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.831873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.831874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.831945 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:16.832226 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:16.832228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:16.832230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:16.832468 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:16.832590 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.832876 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.832958 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:29:16.835891 2902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:29:16.838902 2902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:29:16.853418 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.847100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:41557 (local address 127.0.0.1:36872)
0810 06:29:16.847414 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:16.847458 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:16.847577 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:16.848288 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:16.848296 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:16.848341 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:16.848485 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.848496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.849842 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.849856 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.851861 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.851882 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.852877 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.852880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.852984 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.853019 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 06:29:16.853157 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 06:29:16.853274 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260810 06:29:16.853425 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.847249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.4.73.3:41557)
0810 06:29:16.847630 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.847643 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.847666 (+ 23us) server_negotiation.cc:407] Connection header received
0810 06:29:16.847762 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.847767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.847978 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.848137 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:16.848757 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.849717 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.852004 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.852750 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.852870 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.853012 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.853190 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.853277 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":146,"threads_started":1}
I20260810 06:29:16.853667 3738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.847808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:37167 (local address 127.0.0.1:38524)
0810 06:29:16.848444 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:16.848458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:16.848551 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:16.849263 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:16.849266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:16.849286 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:16.849434 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.849441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.850899 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.850904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.852127 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.852173 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.853298 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.853301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.853401 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.853423 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 06:29:16.853476 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 06:29:16.853539 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":1,"spinlock_wait_cycles":2432,"thread_start_us":105,"threads_started":1}
I20260810 06:29:16.854254 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.848009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.4.73.2:37167)
0810 06:29:16.848796 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.848807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.848829 (+ 22us) server_negotiation.cc:407] Connection header received
0810 06:29:16.848903 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.848907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.849090 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.849267 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:16.849556 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.850719 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.852341 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.853149 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.853837 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.853983 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.854034 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.854117 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":145,"threads_started":1}
I20260810 06:29:16.855010 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.848155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:33133 (local address 127.0.0.1:36254)
0810 06:29:16.849141 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:16.849156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:16.849312 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:16.849977 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:16.849980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:16.850003 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:16.850098 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.850103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.851445 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.851451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.852367 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:16.852378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.853492 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.853495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:16.854682 (+ 1187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.854706 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 06:29:16.854790 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 06:29:16.854858 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":1,"spinlock_wait_cycles":3968,"thread_start_us":86,"threads_started":1}
I20260810 06:29:16.855135 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:16.848614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.4.73.1:33133)
0810 06:29:16.848989 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:16.849003 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:16.849232 (+ 229us) server_negotiation.cc:407] Connection header received
0810 06:29:16.849468 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:16.849472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:16.849663 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:16.849778 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:16.850238 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.851320 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.852490 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:16.853210 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:16.853305 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:16.853430 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:16.854837 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0810 06:29:16.854936 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":101,"threads_started":1}
I20260810 06:29:16.856729 3592 tablet_service.cc:1467] Processing CreateTablet for tablet 3438b8138bdd4918a3ce21d01118fcb7 (DEFAULT_TABLE table=test-table [id=59e87c99d5d540be995483203d70ad6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:16.856892 3370 tablet_service.cc:1467] Processing CreateTablet for tablet 3438b8138bdd4918a3ce21d01118fcb7 (DEFAULT_TABLE table=test-table [id=59e87c99d5d540be995483203d70ad6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:16.857836 3592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3438b8138bdd4918a3ce21d01118fcb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:16.857833 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3438b8138bdd4918a3ce21d01118fcb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:16.857939 3121 tablet_service.cc:1467] Processing CreateTablet for tablet 3438b8138bdd4918a3ce21d01118fcb7 (DEFAULT_TABLE table=test-table [id=59e87c99d5d540be995483203d70ad6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:16.858870 3121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3438b8138bdd4918a3ce21d01118fcb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:16.864228 3744 tablet_bootstrap.cc:492] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Bootstrap starting.
I20260810 06:29:16.865919 3746 tablet_bootstrap.cc:492] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Bootstrap starting.
I20260810 06:29:16.866860 3744 tablet_bootstrap.cc:654] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:16.867605 3744 log.cc:824] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:16.867710 3746 tablet_bootstrap.cc:654] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:16.868414 3748 tablet_bootstrap.cc:492] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109: Bootstrap starting.
I20260810 06:29:16.869087 3746 log.cc:824] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:16.869771 3744 tablet_bootstrap.cc:492] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: No bootstrap required, opened a new log
I20260810 06:29:16.870011 3744 ts_tablet_manager.cc:1397] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:29:16.870484 3748 tablet_bootstrap.cc:654] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:16.871111 3746 tablet_bootstrap.cc:492] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: No bootstrap required, opened a new log
I20260810 06:29:16.871289 3746 ts_tablet_manager.cc:1397] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:29:16.871598 3748 log.cc:824] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:16.873173 3748 tablet_bootstrap.cc:492] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109: No bootstrap required, opened a new log
I20260810 06:29:16.873363 3748 ts_tablet_manager.cc:1397] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:29:16.873939 3744 raft_consensus.cc:348] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:16.874221 3744 raft_consensus.cc:374] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:16.874301 3744 raft_consensus.cc:729] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c1a0a4dd9e460082582e64747a363a, State: Initialized, Role: FOLLOWER
I20260810 06:29:16.874771 3744 consensus_queue.cc:260] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:16.874914 3746 raft_consensus.cc:348] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:16.875211 3746 raft_consensus.cc:374] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:16.875295 3746 raft_consensus.cc:729] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd9e19595194560a9167cc5422ce5f5, State: Initialized, Role: FOLLOWER
I20260810 06:29:16.875792 3744 ts_tablet_manager.cc:1428] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 06:29:16.875799 3746 consensus_queue.cc:260] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:16.876216 3476 heartbeater.cc:503] Master 127.4.73.62:39063 was elected leader, sending a full tablet report...
I20260810 06:29:16.876701 3746 ts_tablet_manager.cc:1428] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260810 06:29:16.876812 3748 raft_consensus.cc:348] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:16.877068 3711 heartbeater.cc:503] Master 127.4.73.62:39063 was elected leader, sending a full tablet report...
I20260810 06:29:16.877130 3748 raft_consensus.cc:374] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:16.877226 3748 raft_consensus.cc:729] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d511ac5936a4409916fd49cf7d6f109, State: Initialized, Role: FOLLOWER
I20260810 06:29:16.877625 3748 consensus_queue.cc:260] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:16.878549 3748 ts_tablet_manager.cc:1428] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 06:29:16.878926 3245 heartbeater.cc:503] Master 127.4.73.62:39063 was elected leader, sending a full tablet report...
I20260810 06:29:17.022346 3754 raft_consensus.cc:482] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:17.022608 3754 raft_consensus.cc:504] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:17.023841 3754 leader_election.cc:288] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76c1a0a4dd9e460082582e64747a363a (127.4.73.2:37167), 8d511ac5936a4409916fd49cf7d6f109 (127.4.73.1:33133)
I20260810 06:29:17.028002 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.024250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:60577 (local address 127.4.73.2:37167)
0810 06:29:17.024413 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:17.024417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:17.024433 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:29:17.024584 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:17.024587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:17.024641 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:17.024713 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:17.025125 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.025766 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.026918 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.027379 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.027463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.027552 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:17.027781 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 06:29:17.027865 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 06:29:17.028023 3739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.024639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:56055 (local address 127.4.73.1:33133)
0810 06:29:17.024769 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:17.024773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:17.024945 (+ 172us) server_negotiation.cc:407] Connection header received
0810 06:29:17.025045 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:17.025049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:17.025105 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:17.025179 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:17.025451 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.025987 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.027032 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.027416 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.027546 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.027610 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:17.027838 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 06:29:17.027919 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 06:29:17.028002 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.024096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:37167 (local address 127.4.73.3:60577)
0810 06:29:17.024282 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:17.024322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:17.024435 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:17.024851 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:17.024853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:17.024878 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:17.025024 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.025032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.025892 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.025896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.026771 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.026780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.027514 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.027519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.027631 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.027656 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:29:17.027729 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 06:29:17.027818 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 06:29:17.028016 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.024493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:33133 (local address 127.4.73.3:56055)
0810 06:29:17.024842 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:17.024854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:17.024950 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:17.025210 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:17.025213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:17.025229 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:17.025341 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.025348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.026097 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.026099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.026906 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.026916 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.027572 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.027577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.027681 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.027699 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:29:17.027802 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:29:17.027874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260810 06:29:17.028748 3155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438b8138bdd4918a3ce21d01118fcb7" candidate_uuid: "abd9e19595194560a9167cc5422ce5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d511ac5936a4409916fd49cf7d6f109" is_pre_election: true
I20260810 06:29:17.028743 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438b8138bdd4918a3ce21d01118fcb7" candidate_uuid: "abd9e19595194560a9167cc5422ce5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c1a0a4dd9e460082582e64747a363a" is_pre_election: true
I20260810 06:29:17.029076 3155 raft_consensus.cc:2454] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd9e19595194560a9167cc5422ce5f5 in term 0.
I20260810 06:29:17.029076 3403 raft_consensus.cc:2454] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abd9e19595194560a9167cc5422ce5f5 in term 0.
I20260810 06:29:17.029579 3510 leader_election.cc:302] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d511ac5936a4409916fd49cf7d6f109, abd9e19595194560a9167cc5422ce5f5; no voters:
I20260810 06:29:17.029812 3754 raft_consensus.cc:2781] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:29:17.029903 3754 raft_consensus.cc:482] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:29:17.029937 3754 raft_consensus.cc:3037] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:17.031080 3754 raft_consensus.cc:504] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:17.031441 3754 leader_election.cc:288] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [CANDIDATE]: Term 1 election: Requested vote from peers 76c1a0a4dd9e460082582e64747a363a (127.4.73.2:37167), 8d511ac5936a4409916fd49cf7d6f109 (127.4.73.1:33133)
I20260810 06:29:17.031845 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438b8138bdd4918a3ce21d01118fcb7" candidate_uuid: "abd9e19595194560a9167cc5422ce5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76c1a0a4dd9e460082582e64747a363a"
I20260810 06:29:17.031868 3155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438b8138bdd4918a3ce21d01118fcb7" candidate_uuid: "abd9e19595194560a9167cc5422ce5f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d511ac5936a4409916fd49cf7d6f109"
I20260810 06:29:17.031980 3155 raft_consensus.cc:3037] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:17.031980 3403 raft_consensus.cc:3037] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:17.033296 3155 raft_consensus.cc:2454] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd9e19595194560a9167cc5422ce5f5 in term 1.
I20260810 06:29:17.033296 3403 raft_consensus.cc:2454] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abd9e19595194560a9167cc5422ce5f5 in term 1.
I20260810 06:29:17.033612 3510 leader_election.cc:302] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d511ac5936a4409916fd49cf7d6f109, abd9e19595194560a9167cc5422ce5f5; no voters:
I20260810 06:29:17.033816 3754 raft_consensus.cc:2781] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:17.034072 3754 raft_consensus.cc:686] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 1 LEADER]: Becoming Leader. State: Replica: abd9e19595194560a9167cc5422ce5f5, State: Running, Role: LEADER
I20260810 06:29:17.034415 3754 consensus_queue.cc:237] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } }
I20260810 06:29:17.037606 2902 catalog_manager.cc:5495] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to abd9e19595194560a9167cc5422ce5f5 (127.4.73.3). New cstate: current_term: 1 leader_uuid: "abd9e19595194560a9167cc5422ce5f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abd9e19595194560a9167cc5422ce5f5" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 } health_report { overall_health: UNKNOWN } } }
W20260810 06:29:17.053020 3712 tablet.cc:2369] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 06:29:17.078507 3477 tablet.cc:2369] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:17.085976 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.082339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.4.73.3:41557)
0810 06:29:17.082479 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:17.082483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:17.082497 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:29:17.082652 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:17.082655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:17.082705 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:17.082805 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:17.083108 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.083612 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.084590 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.084803 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.084888 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.085184 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:17.085545 (+ 361us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:17.085653 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:17.085743 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 06:29:17.085797 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:29:17.091379 3155 raft_consensus.cc:1261] T 3438b8138bdd4918a3ce21d01118fcb7 P 8d511ac5936a4409916fd49cf7d6f109 [term 1 FOLLOWER]: Refusing update from remote peer abd9e19595194560a9167cc5422ce5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:17.091389 3403 raft_consensus.cc:1261] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 1 FOLLOWER]: Refusing update from remote peer abd9e19595194560a9167cc5422ce5f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:17.092053 3754 consensus_queue.cc:1035] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76c1a0a4dd9e460082582e64747a363a" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 37167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:17.092280 3760 consensus_queue.cc:1035] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d511ac5936a4409916fd49cf7d6f109" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:17.099588 3769 mvcc.cc:205] Tried to move back new op lower bound from 7316862390639472640 to 7316862390415790080. Current Snapshot: MvccSnapshot[applied={T|T < 7316862390639472640}]
I20260810 06:29:17.108952 2902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:17.109084 2902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:17.111373 2902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 454f578e2e734f078afe95ca89342188: Sending DeleteTablet for 3 replicas of tablet 3438b8138bdd4918a3ce21d01118fcb7
I20260810 06:29:17.112056 3592 tablet_service.cc:1514] Processing DeleteTablet for tablet 3438b8138bdd4918a3ce21d01118fcb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:17 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:36872
I20260810 06:29:17.112147 3370 tablet_service.cc:1514] Processing DeleteTablet for tablet 3438b8138bdd4918a3ce21d01118fcb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:17 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:38524
I20260810 06:29:17.112216 3121 tablet_service.cc:1514] Processing DeleteTablet for tablet 3438b8138bdd4918a3ce21d01118fcb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:17 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:36254
I20260810 06:29:17.113305 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 3011
I20260810 06:29:17.128527 3787 tablet_replica.cc:330] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: stopping tablet replica
I20260810 06:29:17.128826 3788 tablet_replica.cc:330] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: stopping tablet replica
I20260810 06:29:17.128844 3787 raft_consensus.cc:2229] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:29:17.129154 3788 raft_consensus.cc:2229] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:17.129231 3787 raft_consensus.cc:2258] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:17.129583 3788 raft_consensus.cc:2258] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:17.130821 3787 ts_tablet_manager.cc:1905] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:17.131138 3788 ts_tablet_manager.cc:1905] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:17.133843 3787 ts_tablet_manager.cc:1918] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:17.134034 3787 log.cc:1196] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/3438b8138bdd4918a3ce21d01118fcb7
I20260810 06:29:17.134421 3787 ts_tablet_manager.cc:1939] T 3438b8138bdd4918a3ce21d01118fcb7 P abd9e19595194560a9167cc5422ce5f5: Deleting consensus metadata
I20260810 06:29:17.134606 3788 ts_tablet_manager.cc:1918] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:17.134773 3788 log.cc:1196] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/3438b8138bdd4918a3ce21d01118fcb7
I20260810 06:29:17.135205 3788 ts_tablet_manager.cc:1939] T 3438b8138bdd4918a3ce21d01118fcb7 P 76c1a0a4dd9e460082582e64747a363a: Deleting consensus metadata
I20260810 06:29:17.135675 2867 catalog_manager.cc:4841] TS abd9e19595194560a9167cc5422ce5f5 (127.4.73.3:41557): tablet 3438b8138bdd4918a3ce21d01118fcb7 (table test-table [id=59e87c99d5d540be995483203d70ad6d]) successfully deleted
W20260810 06:29:17.135962 2867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:33133 (error 108)
I20260810 06:29:17.136143 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 3256
I20260810 06:29:17.137651 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.137340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:33133 (local address 127.0.0.1:36268)
0810 06:29:17.137508 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:17.137570 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260810 06:29:17.137835 2867 catalog_manager.cc:4584] TS 8d511ac5936a4409916fd49cf7d6f109 (127.4.73.1:33133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3438b8138bdd4918a3ce21d01118fcb7: Network error: Client connection negotiation failed: client connection to 127.4.73.1:33133: connect: Connection refused (error 111)
I20260810 06:29:17.163439 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 3484
I20260810 06:29:17.172189 3736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.163696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:37167 (local address 127.0.0.1:38538)
0810 06:29:17.171991 (+ 8295us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:17.172067 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.2:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8175}
I20260810 06:29:17.177313 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 2841
2026-08-10T06:29:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2609 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T06:29:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:17Z Disabled control of system clock
I20260810 06:29:17.213850 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43313
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:29:17.318821 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:17.319043 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:17.319077 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:17.322775 3795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:17.322820 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:17.322835 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:17.322851 3795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:17.322865 3795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:17.327222 3795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:17.328080 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:17.329310 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:17.337328 3808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:17.337337 3805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:17.337318 3806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:17.337496 3795 server_base.cc:1034] running on GCE node
I20260810 06:29:17.338115 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:17.339114 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:17.340271 3795 hybrid_clock.cc:648] HybridClock initialized: now 1786343357340246 us; error 37 us; skew 500 ppm
I20260810 06:29:17.342073 3795 webserver.cc:460] Webserver started at http://127.4.73.62:35469/ using document root <none> and password file <none>
I20260810 06:29:17.342639 3795 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:17.342703 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:17.342859 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:17.344779 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "bf5bf46db559418db11dc3c0af183eaf"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "170c5e6c952fcdb06bb693f0cbdcdffa"
server_key_iv: "f579105fbb8b35889b396d41557d33d8"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.345255 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "bf5bf46db559418db11dc3c0af183eaf"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "170c5e6c952fcdb06bb693f0cbdcdffa"
server_key_iv: "f579105fbb8b35889b396d41557d33d8"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.348616 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:17.350798 3818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.351931 3795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:17.352065 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "bf5bf46db559418db11dc3c0af183eaf"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "170c5e6c952fcdb06bb693f0cbdcdffa"
server_key_iv: "f579105fbb8b35889b396d41557d33d8"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.352169 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:17.367398 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:17.367996 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:17.368189 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:17.375092 3795 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43313
I20260810 06:29:17.375103 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43313 every 8 connection(s)
I20260810 06:29:17.375989 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:17.378893 3918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:17.379181 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 3795
I20260810 06:29:17.379311 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:17.379571 4388 external_mini_cluster.cc:1428] Setting key 3d267446bf05e79a419cb9dae1f6f5d0
I20260810 06:29:17.385264 3918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf: Bootstrap starting.
I20260810 06:29:17.387794 3918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:17.388731 3918 log.cc:824] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:17.389667 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.380874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.4.73.62:43313)
0810 06:29:17.381481 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:17.381498 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:17.381538 (+ 40us) server_negotiation.cc:407] Connection header received
0810 06:29:17.382411 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:17.382434 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:17.382910 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:17.383244 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:17.383642 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.385034 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.385943 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.386680 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.386837 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.387594 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:17.387601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:17.387616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:17.388136 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:17.388203 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:17.388231 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:17.388376 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:17.388577 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 06:29:17.388829 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260810 06:29:17.390589 3918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf: No bootstrap required, opened a new log
I20260810 06:29:17.394037 3918 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5bf46db559418db11dc3c0af183eaf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43313 } }
I20260810 06:29:17.394233 3918 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:17.394304 3918 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5bf46db559418db11dc3c0af183eaf, State: Initialized, Role: FOLLOWER
I20260810 06:29:17.394731 3918 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5bf46db559418db11dc3c0af183eaf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43313 } }
I20260810 06:29:17.394867 3918 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:17.394938 3918 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:17.395028 3918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:17.395959 3918 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5bf46db559418db11dc3c0af183eaf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43313 } }
I20260810 06:29:17.396353 3918 leader_election.cc:302] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5bf46db559418db11dc3c0af183eaf; no voters:
I20260810 06:29:17.396697 3918 leader_election.cc:288] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:17.396811 3927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:17.397068 3927 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [term 1 LEADER]: Becoming Leader. State: Replica: bf5bf46db559418db11dc3c0af183eaf, State: Running, Role: LEADER
I20260810 06:29:17.397358 3927 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5bf46db559418db11dc3c0af183eaf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43313 } }
I20260810 06:29:17.397863 3918 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:17.399037 3933 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf5bf46db559418db11dc3c0af183eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5bf46db559418db11dc3c0af183eaf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43313 } } }
I20260810 06:29:17.399175 3933 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:17.399037 3934 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5bf46db559418db11dc3c0af183eaf. Latest consensus state: current_term: 1 leader_uuid: "bf5bf46db559418db11dc3c0af183eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5bf46db559418db11dc3c0af183eaf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43313 } } }
I20260810 06:29:17.399497 3934 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:17.399549 3946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:17.402251 3946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:17.407595 3946 catalog_manager.cc:1347] Generated new cluster ID: 103890a2a9f048b2ac2ea5575e9a96bb
I20260810 06:29:17.407663 3946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:17.427335 3946 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:17.428579 3946 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:17.436487 3946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf: Generated new TSK 0
I20260810 06:29:17.437161 3946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:17.442683 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43313
--builtin_ntp_servers=127.4.73.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:29:17.551286 3960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:17.551505 3960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:17.551537 3960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:17.555315 3960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:17.555357 3960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:17.555428 3960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:29:17.559908 3960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:17.560794 3960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:17.562023 3960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:17.571355 3974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:17.571357 3969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:17.571364 3970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:17.571628 3960 server_base.cc:1034] running on GCE node
I20260810 06:29:17.572052 3960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:17.572649 3960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:17.573808 3960 hybrid_clock.cc:648] HybridClock initialized: now 1786343357573791 us; error 29 us; skew 500 ppm
I20260810 06:29:17.575404 3960 webserver.cc:460] Webserver started at http://127.4.73.1:34243/ using document root <none> and password file <none>
I20260810 06:29:17.575925 3960 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:17.575984 3960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:17.576239 3960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:17.578172 3960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "aaa47ffb6d7b455492e4611f5132f789"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "d1d27fc78cab3c40745dae5fb31899b2"
server_key_iv: "a1710e9db427df88fcc0642690842f07"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.578737 3960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "aaa47ffb6d7b455492e4611f5132f789"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "d1d27fc78cab3c40745dae5fb31899b2"
server_key_iv: "a1710e9db427df88fcc0642690842f07"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.582134 3960 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:17.584486 3984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.585565 3960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:17.585691 3960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "aaa47ffb6d7b455492e4611f5132f789"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "d1d27fc78cab3c40745dae5fb31899b2"
server_key_iv: "a1710e9db427df88fcc0642690842f07"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.585816 3960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:17.607033 3960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:17.607641 3960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:17.607839 3960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:17.608413 3960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:17.609273 3960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:17.609342 3960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.609412 3960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:17.609457 3960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.618184 3960 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:40605
I20260810 06:29:17.618216 4191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:40605 every 8 connection(s)
I20260810 06:29:17.619019 3960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:29:17.619776 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 3960
I20260810 06:29:17.619923 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:29:17.620227 4388 external_mini_cluster.cc:1428] Setting key fbf855eda681166a5e7784759932b398
I20260810 06:29:17.622653 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43313
--builtin_ntp_servers=127.4.73.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:29:17.629696 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.620935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:49675 (local address 127.4.73.62:43313)
0810 06:29:17.621062 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:17.621068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:17.622352 (+ 1284us) server_negotiation.cc:407] Connection header received
0810 06:29:17.624460 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:17.624466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:17.624529 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:17.624636 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:17.625525 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.626243 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.627369 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.627800 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.627946 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.629106 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:17.629112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:17.629116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:17.629133 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:17.629162 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:17.629174 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:17.629253 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:17.629409 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 06:29:17.629489 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:29:17.630362 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.621337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43313 (local address 127.4.73.1:49675)
0810 06:29:17.622190 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:17.622244 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:17.624182 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:17.624817 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:17.624823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:17.624905 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:17.625377 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.625389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.626363 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.626365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.627241 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.627250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.627981 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.627987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.628169 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.628864 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:17.628865 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:17.628900 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:17.628955 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:17.629265 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:17.629272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:17.629275 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 06:29:17.629380 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 06:29:17.629578 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":142,"threads_started":1}
I20260810 06:29:17.631614 4192 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43313
I20260810 06:29:17.631883 4192 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:17.632543 4192 heartbeater.cc:511] Master 127.4.73.62:43313 requested a full tablet report, sending...
I20260810 06:29:17.634140 3852 ts_manager.cc:194] Registered new tserver with Master: aaa47ffb6d7b455492e4611f5132f789 (127.4.73.1:40605)
I20260810 06:29:17.635263 3852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.1:49675
W20260810 06:29:17.744809 4196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:17.745085 4196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:17.745137 4196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:17.749718 4196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:17.749792 4196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:17.749874 4196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:29:17.754323 4196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:17.755492 4196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:17.757019 4196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:17.767808 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:17.767969 4196 server_base.cc:1034] running on GCE node
W20260810 06:29:17.767815 4214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:17.768184 4210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:17.768620 4196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:17.769138 4196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:17.770296 4196 hybrid_clock.cc:648] HybridClock initialized: now 1786343357770279 us; error 28 us; skew 500 ppm
I20260810 06:29:17.772033 4196 webserver.cc:460] Webserver started at http://127.4.73.2:45887/ using document root <none> and password file <none>
I20260810 06:29:17.772629 4196 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:17.772693 4196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:17.772925 4196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:17.774889 4196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "a887163bf2184f0d83539350eb33503d"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "a5c6f74ce7d0012e4acf00f5888871a6"
server_key_iv: "68bbb0ca48bc1bcdb3eb9db56d669bcd"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.775437 4196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "a887163bf2184f0d83539350eb33503d"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "a5c6f74ce7d0012e4acf00f5888871a6"
server_key_iv: "68bbb0ca48bc1bcdb3eb9db56d669bcd"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.778978 4196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 06:29:17.781337 4221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.782459 4196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:29:17.782593 4196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "a887163bf2184f0d83539350eb33503d"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "a5c6f74ce7d0012e4acf00f5888871a6"
server_key_iv: "68bbb0ca48bc1bcdb3eb9db56d669bcd"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.782697 4196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:17.804502 4196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:17.805049 4196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:17.805222 4196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:17.805781 4196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:17.806663 4196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:17.806735 4196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.806805 4196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:17.806849 4196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.815708 4196 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:46745
I20260810 06:29:17.815747 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:46745 every 8 connection(s)
I20260810 06:29:17.816540 4196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:29:17.823056 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 4196
I20260810 06:29:17.823186 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:29:17.823490 4388 external_mini_cluster.cc:1428] Setting key 8fecdd66cdfa2b0460e52adfa2a25b8c
I20260810 06:29:17.825330 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.818158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:33111 (local address 127.4.73.62:43313)
0810 06:29:17.818321 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:17.818326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:17.819258 (+ 932us) server_negotiation.cc:407] Connection header received
0810 06:29:17.820497 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:17.820500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:17.820543 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:17.820612 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:17.821395 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.822000 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.823265 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.823495 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.823616 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.824851 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:17.824856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:17.824860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:17.824892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:17.824918 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:17.824929 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:17.825006 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:17.825176 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 06:29:17.825225 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 06:29:17.825647 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43313
--builtin_ntp_servers=127.4.73.20:42949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:29:17.826377 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:17.818525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43313 (local address 127.4.73.2:33111)
0810 06:29:17.819117 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:17.819153 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:17.820292 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:17.820766 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:17.820772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:17.820847 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:17.821278 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.821290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.822119 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.822121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.823137 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:17.823150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:17.823679 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:17.823684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:17.823870 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:17.824566 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:17.824568 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:17.824604 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:17.824661 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:17.825025 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:17.825034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:17.825039 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 06:29:17.825166 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 06:29:17.825441 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260810 06:29:17.827521 4431 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43313
I20260810 06:29:17.827769 4431 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:17.828389 4431 heartbeater.cc:511] Master 127.4.73.62:43313 requested a full tablet report, sending...
I20260810 06:29:17.829537 3852 ts_manager.cc:194] Registered new tserver with Master: a887163bf2184f0d83539350eb33503d (127.4.73.2:46745)
I20260810 06:29:17.830343 3852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.2:33111
W20260810 06:29:17.933256 4434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:17.933491 4434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:17.933542 4434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:17.937427 4434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:17.937475 4434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:17.937526 4434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:29:17.941919 4434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:17.942770 4434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:17.943996 4434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:17.952847 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:17.952890 4446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:17.952934 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:17.953007 4434 server_base.cc:1034] running on GCE node
I20260810 06:29:17.953459 4434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:17.954025 4434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:17.955204 4434 hybrid_clock.cc:648] HybridClock initialized: now 1786343357955161 us; error 47 us; skew 500 ppm
I20260810 06:29:17.957075 4434 webserver.cc:460] Webserver started at http://127.4.73.3:43127/ using document root <none> and password file <none>
I20260810 06:29:17.957625 4434 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:17.957686 4434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:17.957928 4434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:17.959800 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "76b20702a0be46f291f613450a869028"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "69fd43d971831131a86433ac0ef8ef1b"
server_key_iv: "681d50316130009a8fb206d4802ae47c"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.960428 4434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "76b20702a0be46f291f613450a869028"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "69fd43d971831131a86433ac0ef8ef1b"
server_key_iv: "681d50316130009a8fb206d4802ae47c"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.964022 4434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:29:17.966331 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.967468 4434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:17.967602 4434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "76b20702a0be46f291f613450a869028"
format_stamp: "Formatted at 2026-08-10 06:29:17 on dist-test-slave-45dx"
server_key: "69fd43d971831131a86433ac0ef8ef1b"
server_key_iv: "681d50316130009a8fb206d4802ae47c"
server_key_version: "encryptionkey@0"
I20260810 06:29:17.967712 4434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:17.987975 4434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:17.988644 4434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:17.988891 4434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:17.989435 4434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:17.990372 4434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:17.990451 4434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.990522 4434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:17.990559 4434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:17.999073 4434 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:41095
I20260810 06:29:17.999189 4641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:41095 every 8 connection(s)
I20260810 06:29:17.999915 4434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:29:18.003162 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 4434
I20260810 06:29:18.003252 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:29:18.003531 4388 external_mini_cluster.cc:1428] Setting key 43d769f35ba93b1b824e198624d2c531
I20260810 06:29:18.009678 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.001664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:57125 (local address 127.4.73.62:43313)
0810 06:29:18.001829 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.001833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.002859 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 06:29:18.004615 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.004618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.004675 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.004757 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:18.005800 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.006509 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.007652 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.007902 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.008021 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.009169 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:18.009173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:18.009176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:18.009189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:18.009212 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:18.009229 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:18.009310 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.009476 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.009539 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 06:29:18.010473 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.002074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:43313 (local address 127.4.73.3:57125)
0810 06:29:18.002704 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:18.002743 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:18.004296 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:18.004932 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:18.004940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:18.005022 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:18.005615 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.005630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.006642 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.006650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.007521 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.007530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.008057 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.008061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.008226 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.008911 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:18.008912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:18.008952 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:18.009014 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:18.009312 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:18.009325 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:18.009334 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 06:29:18.009473 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 06:29:18.009744 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260810 06:29:18.011520 4645 heartbeater.cc:347] Connected to a master server at 127.4.73.62:43313
I20260810 06:29:18.011754 4645 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:18.012185 4645 heartbeater.cc:511] Master 127.4.73.62:43313 requested a full tablet report, sending...
I20260810 06:29:18.013357 3852 ts_manager.cc:194] Registered new tserver with Master: 76b20702a0be46f291f613450a869028 (127.4.73.3:41095)
I20260810 06:29:18.013964 3852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.3:57125
I20260810 06:29:18.017624 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:29:18.023511 3925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.019502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.4.73.62:43313)
0810 06:29:18.019659 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.019664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.019817 (+ 153us) server_negotiation.cc:407] Connection header received
0810 06:29:18.019958 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.019961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.020002 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.020091 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:18.020413 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.020897 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.021719 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.021979 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.022086 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.023021 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:18.023025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:18.023027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:18.023039 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:18.023067 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:18.023075 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:18.023202 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.023329 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.023366 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:29:18.026048 3852 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:29:18.028959 3852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:29:18.043074 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.037125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:46745 (local address 127.0.0.1:59054)
0810 06:29:18.038217 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:18.038233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:18.038322 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:18.038809 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:18.038815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:18.038867 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:18.038998 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.039006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.040195 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.040200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.041523 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.041533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.042520 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.042523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.042641 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.042686 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 06:29:18.042804 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 06:29:18.042929 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":45,"thread_start_us":138,"threads_started":1}
I20260810 06:29:18.043136 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.037692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.4.73.2:46745)
0810 06:29:18.038344 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.038357 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.038387 (+ 30us) server_negotiation.cc:407] Connection header received
0810 06:29:18.038492 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.038497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.038666 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.038819 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:18.039133 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.040078 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.041653 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.042388 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.042503 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.042701 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.042854 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.042968 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":116,"threads_started":1}
I20260810 06:29:18.043558 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.036855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:41095 (local address 127.0.0.1:38148)
0810 06:29:18.037916 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:18.037964 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:18.038329 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:18.038810 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:18.038815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:18.038989 (+ 174us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:18.039079 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.039086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.040519 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.040523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.041841 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.041859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.043120 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.043123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.043219 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.043242 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 06:29:18.043313 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 06:29:18.043410 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":53,"spinlock_wait_cycles":319360,"thread_start_us":82,"threads_started":1}
I20260810 06:29:18.043593 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.037579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40605 (local address 127.0.0.1:41380)
0810 06:29:18.038259 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:18.038276 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:18.038380 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:18.039453 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:18.039456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:18.039474 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:18.039556 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.039561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.040923 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.040927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.042026 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.042034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.043151 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.043156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.043265 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.043294 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 06:29:18.043368 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 06:29:18.043446 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":380,"thread_start_us":102,"threads_started":1}
I20260810 06:29:18.043998 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.037766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.4.73.1:40605)
0810 06:29:18.039021 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.039030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.039047 (+ 17us) server_negotiation.cc:407] Connection header received
0810 06:29:18.039105 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.039108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.039286 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.039448 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:18.039674 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.040798 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.042137 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.042804 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.042911 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.043080 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.043733 (+ 653us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.043826 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":147,"threads_started":1}
I20260810 06:29:18.044088 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.037039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.4.73.3:41095)
0810 06:29:18.037445 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.037458 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.038293 (+ 835us) server_negotiation.cc:407] Connection header received
0810 06:29:18.038534 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.038538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.038676 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.038827 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:18.039191 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.040379 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.041974 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.042755 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.042882 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.043053 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.043818 (+ 765us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.043917 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":176,"threads_started":1}
I20260810 06:29:18.046350 4302 tablet_service.cc:1467] Processing CreateTablet for tablet 84c9edbb0bf24074af224d59a293ad42 (DEFAULT_TABLE table=test-table [id=0de6262c44c3482391da626e5b409187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:18.047348 4302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c9edbb0bf24074af224d59a293ad42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:18.047195 4535 tablet_service.cc:1467] Processing CreateTablet for tablet 84c9edbb0bf24074af224d59a293ad42 (DEFAULT_TABLE table=test-table [id=0de6262c44c3482391da626e5b409187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:18.047194 4078 tablet_service.cc:1467] Processing CreateTablet for tablet 84c9edbb0bf24074af224d59a293ad42 (DEFAULT_TABLE table=test-table [id=0de6262c44c3482391da626e5b409187]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:18.048199 4535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c9edbb0bf24074af224d59a293ad42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:18.048200 4078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84c9edbb0bf24074af224d59a293ad42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:18.054045 4671 tablet_bootstrap.cc:492] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Bootstrap starting.
I20260810 06:29:18.055171 4673 tablet_bootstrap.cc:492] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: Bootstrap starting.
I20260810 06:29:18.056468 4671 tablet_bootstrap.cc:654] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:18.056495 4675 tablet_bootstrap.cc:492] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Bootstrap starting.
I20260810 06:29:18.057317 4673 tablet_bootstrap.cc:654] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:18.057343 4671 log.cc:824] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:18.058077 4673 log.cc:824] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:18.058372 4675 tablet_bootstrap.cc:654] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:18.059208 4675 log.cc:824] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:18.059607 4671 tablet_bootstrap.cc:492] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: No bootstrap required, opened a new log
I20260810 06:29:18.059809 4671 ts_tablet_manager.cc:1397] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260810 06:29:18.060877 4675 tablet_bootstrap.cc:492] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: No bootstrap required, opened a new log
I20260810 06:29:18.061057 4675 ts_tablet_manager.cc:1397] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:29:18.063038 4671 raft_consensus.cc:348] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.063279 4671 raft_consensus.cc:374] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:18.063340 4671 raft_consensus.cc:729] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a887163bf2184f0d83539350eb33503d, State: Initialized, Role: FOLLOWER
I20260810 06:29:18.063799 4671 consensus_queue.cc:260] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.064450 4431 heartbeater.cc:503] Master 127.4.73.62:43313 was elected leader, sending a full tablet report...
I20260810 06:29:18.064365 4675 raft_consensus.cc:348] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.064601 4675 raft_consensus.cc:374] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:18.064661 4675 raft_consensus.cc:729] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b20702a0be46f291f613450a869028, State: Initialized, Role: FOLLOWER
I20260810 06:29:18.064760 4671 ts_tablet_manager.cc:1428] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260810 06:29:18.065203 4673 tablet_bootstrap.cc:492] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: No bootstrap required, opened a new log
I20260810 06:29:18.065137 4675 consensus_queue.cc:260] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.065389 4673 ts_tablet_manager.cc:1397] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260810 06:29:18.066094 4645 heartbeater.cc:503] Master 127.4.73.62:43313 was elected leader, sending a full tablet report...
I20260810 06:29:18.066541 4675 ts_tablet_manager.cc:1428] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260810 06:29:18.067371 4432 tablet.cc:2369] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:18.069155 4673 raft_consensus.cc:348] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.069402 4673 raft_consensus.cc:374] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:18.069468 4673 raft_consensus.cc:729] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa47ffb6d7b455492e4611f5132f789, State: Initialized, Role: FOLLOWER
I20260810 06:29:18.069805 4673 consensus_queue.cc:260] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.070370 4192 heartbeater.cc:503] Master 127.4.73.62:43313 was elected leader, sending a full tablet report...
I20260810 06:29:18.070587 4673 ts_tablet_manager.cc:1428] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260810 06:29:18.120074 4193 tablet.cc:2369] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:18.200233 4684 raft_consensus.cc:482] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:18.200489 4684 raft_consensus.cc:504] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.201558 4684 leader_election.cc:288] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76b20702a0be46f291f613450a869028 (127.4.73.3:41095), aaa47ffb6d7b455492e4611f5132f789 (127.4.73.1:40605)
I20260810 06:29:18.206317 4668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.202253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:37587 (local address 127.4.73.1:40605)
0810 06:29:18.202380 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.202385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.202540 (+ 155us) server_negotiation.cc:407] Connection header received
0810 06:29:18.202673 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.202676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.202728 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.202803 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:18.203045 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.203714 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.205067 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.205648 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.205760 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.205820 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.206122 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.206192 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:29:18.206317 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.201963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:46353 (local address 127.4.73.3:41095)
0810 06:29:18.202102 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.202106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.202143 (+ 37us) server_negotiation.cc:407] Connection header received
0810 06:29:18.202203 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.202205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.202251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.202317 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:18.202711 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.203524 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.205055 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.205651 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.205760 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.205820 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.206122 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.206194 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 06:29:18.206322 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.201822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:41095 (local address 127.4.73.2:46353)
0810 06:29:18.201958 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:18.201971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:18.202086 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:18.202436 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:18.202440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:18.202464 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:18.202600 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.202609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.203630 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.203632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.204916 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.204928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.205774 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.205777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.205974 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.206009 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 06:29:18.206087 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 06:29:18.206167 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":121}
I20260810 06:29:18.206337 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.202115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40605 (local address 127.4.73.2:37587)
0810 06:29:18.202444 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:18.202457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:18.202549 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:18.202822 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:18.202825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:18.202840 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:18.202933 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.202937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.203804 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.203807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.204956 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:18.204967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.205774 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.205777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:18.205962 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.206008 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 06:29:18.206087 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 06:29:18.206167 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":56,"thread_start_us":133,"threads_started":1}
I20260810 06:29:18.207078 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c9edbb0bf24074af224d59a293ad42" candidate_uuid: "a887163bf2184f0d83539350eb33503d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b20702a0be46f291f613450a869028" is_pre_election: true
I20260810 06:29:18.207067 4110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c9edbb0bf24074af224d59a293ad42" candidate_uuid: "a887163bf2184f0d83539350eb33503d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa47ffb6d7b455492e4611f5132f789" is_pre_election: true
I20260810 06:29:18.207363 4110 raft_consensus.cc:2454] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a887163bf2184f0d83539350eb33503d in term 0.
I20260810 06:29:18.207363 4568 raft_consensus.cc:2454] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a887163bf2184f0d83539350eb33503d in term 0.
I20260810 06:29:18.207849 4229 leader_election.cc:302] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76b20702a0be46f291f613450a869028, a887163bf2184f0d83539350eb33503d; no voters:
I20260810 06:29:18.208086 4684 raft_consensus.cc:2781] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:29:18.208168 4684 raft_consensus.cc:482] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:29:18.208199 4684 raft_consensus.cc:3037] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:18.209233 4684 raft_consensus.cc:504] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.209584 4684 leader_election.cc:288] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [CANDIDATE]: Term 1 election: Requested vote from peers 76b20702a0be46f291f613450a869028 (127.4.73.3:41095), aaa47ffb6d7b455492e4611f5132f789 (127.4.73.1:40605)
I20260810 06:29:18.210002 4110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c9edbb0bf24074af224d59a293ad42" candidate_uuid: "a887163bf2184f0d83539350eb33503d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa47ffb6d7b455492e4611f5132f789"
I20260810 06:29:18.210002 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84c9edbb0bf24074af224d59a293ad42" candidate_uuid: "a887163bf2184f0d83539350eb33503d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76b20702a0be46f291f613450a869028"
I20260810 06:29:18.210126 4110 raft_consensus.cc:3037] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:18.210126 4568 raft_consensus.cc:3037] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:18.211372 4110 raft_consensus.cc:2454] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a887163bf2184f0d83539350eb33503d in term 1.
I20260810 06:29:18.211386 4568 raft_consensus.cc:2454] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a887163bf2184f0d83539350eb33503d in term 1.
I20260810 06:29:18.211696 4229 leader_election.cc:302] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a887163bf2184f0d83539350eb33503d, aaa47ffb6d7b455492e4611f5132f789; no voters:
I20260810 06:29:18.211891 4684 raft_consensus.cc:2781] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:18.212121 4684 raft_consensus.cc:686] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 1 LEADER]: Becoming Leader. State: Replica: a887163bf2184f0d83539350eb33503d, State: Running, Role: LEADER
I20260810 06:29:18.212473 4684 consensus_queue.cc:237] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } }
I20260810 06:29:18.215562 3852 catalog_manager.cc:5495] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d reported cstate change: term changed from 0 to 1, leader changed from <none> to a887163bf2184f0d83539350eb33503d (127.4.73.2). New cstate: current_term: 1 leader_uuid: "a887163bf2184f0d83539350eb33503d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a887163bf2184f0d83539350eb33503d" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 46745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 } health_report { overall_health: UNKNOWN } } }
I20260810 06:29:18.229182 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.225787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.4.73.2:46745)
0810 06:29:18.225899 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:18.225903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:18.225915 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:29:18.226055 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:18.226057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:18.226101 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:18.226194 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:18.226430 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.226898 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.227760 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:18.227974 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:18.228080 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:18.228433 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:18.228774 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:18.228924 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:18.229019 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 06:29:18.229050 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:29:18.234035 4568 raft_consensus.cc:1261] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 1 FOLLOWER]: Refusing update from remote peer a887163bf2184f0d83539350eb33503d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:18.234076 4110 raft_consensus.cc:1261] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 1 FOLLOWER]: Refusing update from remote peer a887163bf2184f0d83539350eb33503d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:18.234670 4684 consensus_queue.cc:1035] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [LEADER]: Connected to new peer: Peer: permanent_uuid: "76b20702a0be46f291f613450a869028" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:18.234834 4689 consensus_queue.cc:1035] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa47ffb6d7b455492e4611f5132f789" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:18.241583 4696 mvcc.cc:205] Tried to move back new op lower bound from 7316862395321020416 to 7316862395241271296. Current Snapshot: MvccSnapshot[applied={T|T < 7316862395321020416}]
I20260810 06:29:18.248991 3852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:18.249148 3852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:18.251370 3852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf5bf46db559418db11dc3c0af183eaf: Sending DeleteTablet for 3 replicas of tablet 84c9edbb0bf24074af224d59a293ad42
I20260810 06:29:18.252076 4078 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c9edbb0bf24074af224d59a293ad42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:18 UTC) from {username='slave'} at 127.0.0.1:41380
I20260810 06:29:18.252077 4535 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c9edbb0bf24074af224d59a293ad42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:18 UTC) from {username='slave'} at 127.0.0.1:38148
I20260810 06:29:18.252177 4302 tablet_service.cc:1514] Processing DeleteTablet for tablet 84c9edbb0bf24074af224d59a293ad42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:18 UTC) from {username='slave'} at 127.0.0.1:59054
I20260810 06:29:18.252521 4713 tablet_replica.cc:330] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: stopping tablet replica
I20260810 06:29:18.252529 4712 tablet_replica.cc:330] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789: stopping tablet replica
I20260810 06:29:18.252779 4712 raft_consensus.cc:2229] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:18.252777 4713 raft_consensus.cc:2229] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:18.253048 4713 raft_consensus.cc:2258] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:18.253057 4712 raft_consensus.cc:2258] T 84c9edbb0bf24074af224d59a293ad42 P aaa47ffb6d7b455492e4611f5132f789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:18.253161 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 3960
I20260810 06:29:18.269121 4714 tablet_replica.cc:330] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: stopping tablet replica
I20260810 06:29:18.269492 4714 raft_consensus.cc:2229] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:29:18.269909 4714 raft_consensus.cc:2258] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:18.273532 4714 ts_tablet_manager.cc:1905] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:18.273532 4713 ts_tablet_manager.cc:1905] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:18.276283 4714 ts_tablet_manager.cc:1918] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:18.276283 4713 ts_tablet_manager.cc:1918] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:18.276417 4713 log.cc:1196] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/84c9edbb0bf24074af224d59a293ad42
I20260810 06:29:18.276417 4714 log.cc:1196] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/84c9edbb0bf24074af224d59a293ad42
I20260810 06:29:18.276758 4713 ts_tablet_manager.cc:1939] T 84c9edbb0bf24074af224d59a293ad42 P 76b20702a0be46f291f613450a869028: Deleting consensus metadata
I20260810 06:29:18.276775 4714 ts_tablet_manager.cc:1939] T 84c9edbb0bf24074af224d59a293ad42 P a887163bf2184f0d83539350eb33503d: Deleting consensus metadata
I20260810 06:29:18.277979 3830 catalog_manager.cc:4841] TS 76b20702a0be46f291f613450a869028 (127.4.73.3:41095): tablet 84c9edbb0bf24074af224d59a293ad42 (table test-table [id=0de6262c44c3482391da626e5b409187]) successfully deleted
I20260810 06:29:18.277984 3821 catalog_manager.cc:4841] TS a887163bf2184f0d83539350eb33503d (127.4.73.2:46745): tablet 84c9edbb0bf24074af224d59a293ad42 (table test-table [id=0de6262c44c3482391da626e5b409187]) successfully deleted
W20260810 06:29:18.280289 3830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:40605 (error 108)
I20260810 06:29:18.280460 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 4196
I20260810 06:29:18.281599 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:18.281194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:40605 (local address 127.0.0.1:41394)
0810 06:29:18.281403 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:18.281491 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260810 06:29:18.281862 3830 catalog_manager.cc:4584] TS aaa47ffb6d7b455492e4611f5132f789 (127.4.73.1:40605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84c9edbb0bf24074af224d59a293ad42: Network error: Client connection negotiation failed: client connection to 127.4.73.1:40605: connect: Connection refused (error 111)
I20260810 06:29:18.292049 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 4434
I20260810 06:29:18.312050 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 3795
2026-08-10T06:29:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1133 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:18 dist-test-slave-45dx krb5kdc[4722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:18 dist-test-slave-45dx krb5kdc[4722](info): set up 2 sockets
Aug 10 06:29:18 dist-test-slave-45dx krb5kdc[4722](info): commencing operation
krb5kdc: starting...
W20260810 06:29:19.541191 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:19 dist-test-slave-45dx krb5kdc[4722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:19.666716 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:45145
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:45145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:19.775882 4741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:19.776122 4741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:19.776160 4741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:19.780006 4741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:19.780061 4741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:19.780079 4741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:19.780097 4741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:19.780112 4741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:19.785236 4741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:45145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:45145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:19.786142 4741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:19.787060 4741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:19.795435 4751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:19.795497 4748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:19.795537 4741 server_base.cc:1034] running on GCE node
W20260810 06:29:19.795446 4756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:19.796403 4741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:19.797413 4741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:19.798602 4741 hybrid_clock.cc:648] HybridClock initialized: now 1786343359798578 us; error 45 us; skew 500 ppm
Aug 10 06:29:19 dist-test-slave-45dx krb5kdc[4722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343359, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:19.801302 4741 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:19.802205 4741 webserver.cc:460] Webserver started at http://127.4.73.62:44123/ using document root <none> and password file <none>
I20260810 06:29:19.802733 4741 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:19.802825 4741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:19.803059 4741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:19.805074 4741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "9bd2a27409944751bffd286a0e9c26e6"
format_stamp: "Formatted at 2026-08-10 06:29:19 on dist-test-slave-45dx"
server_key: "c349a7d7490d034b3f51f0b8d2fa143b"
server_key_iv: "08a9ba746cd825d0007a5a96dffe2edc"
server_key_version: "encryptionkey@0"
I20260810 06:29:19.805655 4741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "9bd2a27409944751bffd286a0e9c26e6"
format_stamp: "Formatted at 2026-08-10 06:29:19 on dist-test-slave-45dx"
server_key: "c349a7d7490d034b3f51f0b8d2fa143b"
server_key_iv: "08a9ba746cd825d0007a5a96dffe2edc"
server_key_version: "encryptionkey@0"
I20260810 06:29:19.809291 4741 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:19.811596 4768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:19.812842 4741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:19.813006 4741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "9bd2a27409944751bffd286a0e9c26e6"
format_stamp: "Formatted at 2026-08-10 06:29:19 on dist-test-slave-45dx"
server_key: "c349a7d7490d034b3f51f0b8d2fa143b"
server_key_iv: "08a9ba746cd825d0007a5a96dffe2edc"
server_key_version: "encryptionkey@0"
I20260810 06:29:19.813125 4741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:19.838833 4741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:19.841687 4741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:19.841922 4741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:19.848907 4741 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:45145
I20260810 06:29:19.848929 4865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:45145 every 8 connection(s)
I20260810 06:29:19.849831 4741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:19.852602 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:19.852777 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 4741
I20260810 06:29:19.852923 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:19.853216 4388 external_mini_cluster.cc:1428] Setting key e9638dfd63272961157bda92f8d03e11
I20260810 06:29:19.859603 4866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6: Bootstrap starting.
I20260810 06:29:19.861914 4866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6: Neither blocks nor log segments found. Creating new log.
Aug 10 06:29:19 dist-test-slave-45dx krb5kdc[4722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:19.862715 4866 log.cc:824] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:19.864540 4866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6: No bootstrap required, opened a new log
I20260810 06:29:19.867527 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:19.854598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.4.73.62:45145)
0810 06:29:19.855046 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:19.855064 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:19.855107 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:29:19.855862 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:19.855893 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:19.856319 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:19.856699 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:19.857377 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:19.858907 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:19.859742 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:19.860202 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:19.860411 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:19.863016 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:19.863021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:19.863033 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:19.863059 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:19.864856 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:19.865285 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:19.865290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:19.865295 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:19.865429 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:19.865717 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:19.865721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:19.865722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:19.866087 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:19.866286 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:19.866566 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 06:29:19.866844 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":130,"threads_started":1}
I20260810 06:29:19.867992 4866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd2a27409944751bffd286a0e9c26e6" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45145 } }
I20260810 06:29:19.868227 4866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:19.868338 4866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bd2a27409944751bffd286a0e9c26e6, State: Initialized, Role: FOLLOWER
I20260810 06:29:19.868796 4866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd2a27409944751bffd286a0e9c26e6" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45145 } }
I20260810 06:29:19.868920 4866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:19.869002 4866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:19.869113 4866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:19.870081 4866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd2a27409944751bffd286a0e9c26e6" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45145 } }
I20260810 06:29:19.870430 4866 leader_election.cc:302] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bd2a27409944751bffd286a0e9c26e6; no voters:
I20260810 06:29:19.870739 4866 leader_election.cc:288] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:19.870858 4876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:19.871100 4876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9bd2a27409944751bffd286a0e9c26e6, State: Running, Role: LEADER
I20260810 06:29:19.871452 4876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd2a27409944751bffd286a0e9c26e6" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45145 } }
I20260810 06:29:19.871882 4866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:19.873118 4877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bd2a27409944751bffd286a0e9c26e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd2a27409944751bffd286a0e9c26e6" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45145 } } }
I20260810 06:29:19.873126 4878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bd2a27409944751bffd286a0e9c26e6. Latest consensus state: current_term: 1 leader_uuid: "9bd2a27409944751bffd286a0e9c26e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bd2a27409944751bffd286a0e9c26e6" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 45145 } } }
I20260810 06:29:19.873298 4878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:19.873586 4877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:19.874595 4893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:19.877066 4893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:19.882450 4893 catalog_manager.cc:1347] Generated new cluster ID: 0acc9d1f8aaa40879a162a9fca617e1d
I20260810 06:29:19.882515 4893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:19.908721 4893 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:19.909579 4893 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:19.920218 4893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6: Generated new TSK 0
I20260810 06:29:19.920883 4893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.73.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
Entry for principal kudu/127.4.73.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab.
I20260810 06:29:19.950726 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45145
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:20.061638 4912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:20.062114 4912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:20.062211 4912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:20.066527 4912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:20.066605 4912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:20.066697 4912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:29:20.072021 4912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.1.keytab
--principal=kudu/127.4.73.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:20.073536 4912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:20.074586 4912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:20.084391 4924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:20.084553 4912 server_base.cc:1034] running on GCE node
W20260810 06:29:20.084510 4922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:20.084388 4921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:20.085119 4912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:20.085645 4912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:20.086796 4912 hybrid_clock.cc:648] HybridClock initialized: now 1786343360086767 us; error 42 us; skew 500 ppm
Aug 10 06:29:20 dist-test-slave-45dx krb5kdc[4722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343360, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:20.090143 4912 init.cc:377] Logged in from keytab as kudu/127.4.73.1@KRBTEST.COM (short username kudu)
I20260810 06:29:20.091244 4912 webserver.cc:460] Webserver started at http://127.4.73.1:39941/ using document root <none> and password file <none>
I20260810 06:29:20.091954 4912 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:20.092051 4912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:20.092293 4912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:20.094293 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "8cd69950bf834dcfa29ba71d4c31b58e"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "a164d3c8921974d9cc76ed1c2cbf7c48"
server_key_iv: "47af2753081dc14993aa01c5cd8efa8a"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.094801 4912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "8cd69950bf834dcfa29ba71d4c31b58e"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "a164d3c8921974d9cc76ed1c2cbf7c48"
server_key_iv: "47af2753081dc14993aa01c5cd8efa8a"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.098322 4912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:20.100646 4935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.101826 4912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:29:20.101926 4912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "8cd69950bf834dcfa29ba71d4c31b58e"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "a164d3c8921974d9cc76ed1c2cbf7c48"
server_key_iv: "47af2753081dc14993aa01c5cd8efa8a"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.102010 4912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:20.121919 4912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:20.124792 4912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:20.124954 4912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:20.125496 4912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:20.126267 4912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:20.126315 4912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.126354 4912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:20.126369 4912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.135279 4912 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:45209
I20260810 06:29:20.135305 5129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:45209 every 8 connection(s)
I20260810 06:29:20.136169 4912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:29:20.137104 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 4912
I20260810 06:29:20.137281 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:29:20.137598 4388 external_mini_cluster.cc:1428] Setting key 8b4ef9e2b8335ef3e65cc73606955662
Aug 10 06:29:20 dist-test-slave-45dx krb5kdc[4722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343360, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.1@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.2@KRBTEST.COM" created.
I20260810 06:29:20.151911 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.139137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:37219 (local address 127.4.73.62:45145)
0810 06:29:20.139272 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.139275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.140354 (+ 1079us) server_negotiation.cc:407] Connection header received
0810 06:29:20.141722 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.141726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.141797 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.141880 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:20.143032 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.143582 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.144951 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.145192 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.145310 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.147811 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:20.147817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:20.147824 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:20.147850 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:20.149582 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.150247 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.150251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.150253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.150348 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.150776 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.150781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.150783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.150984 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:20.151095 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.151597 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.151729 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:29:20.153075 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.139698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45145 (local address 127.4.73.1:37219)
0810 06:29:20.140185 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.140230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.141519 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.142058 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.142065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.142414 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:20.142882 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.142896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.143733 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.143737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.144813 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.144824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.145349 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.145354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.145530 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.146159 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:20.146176 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:20.147632 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:20.149713 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:20.149719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:20.149733 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:20.150140 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:20.150503 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:20.150506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:20.150508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:20.150686 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:20.151095 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:20.151103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:20.151357 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.151553 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.151892 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":138,"threads_started":1}
I20260810 06:29:20.154419 5130 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45145
I20260810 06:29:20.154686 5130 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:20.155323 5130 heartbeater.cc:511] Master 127.4.73.62:45145 requested a full tablet report, sending...
I20260810 06:29:20.157191 4797 ts_manager.cc:194] Registered new tserver with Master: 8cd69950bf834dcfa29ba71d4c31b58e (127.4.73.1:45209)
I20260810 06:29:20.158897 4797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.1@KRBTEST.COM'} at 127.4.73.1:37219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
Entry for principal kudu/127.4.73.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab.
I20260810 06:29:20.167479 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45145
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:20.280975 5137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:20.281211 5137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:20.281275 5137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:20.285239 5137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:20.285291 5137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:20.285389 5137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:29:20.290601 5137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.2.keytab
--principal=kudu/127.4.73.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:20.291565 5137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:20.292670 5137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:20.302343 5151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:20.302338 5155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:20.302386 5137 server_base.cc:1034] running on GCE node
W20260810 06:29:20.302341 5150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:20.302994 5137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:20.303656 5137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:20.304824 5137 hybrid_clock.cc:648] HybridClock initialized: now 1786343360304811 us; error 25 us; skew 500 ppm
Aug 10 06:29:20 dist-test-slave-45dx krb5kdc[4722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343360, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:20.307514 5137 init.cc:377] Logged in from keytab as kudu/127.4.73.2@KRBTEST.COM (short username kudu)
I20260810 06:29:20.308384 5137 webserver.cc:460] Webserver started at http://127.4.73.2:33451/ using document root <none> and password file <none>
I20260810 06:29:20.308934 5137 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:20.309046 5137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:20.309271 5137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:20.311267 5137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "ad7469d11ca74242a29e9bb97b6d19f2"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "43c5c37812a0f4de8621b09c74d4b79c"
server_key_iv: "c5c4e395f6727c43aeb5f583b56a172a"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.311849 5137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "ad7469d11ca74242a29e9bb97b6d19f2"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "43c5c37812a0f4de8621b09c74d4b79c"
server_key_iv: "c5c4e395f6727c43aeb5f583b56a172a"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.315465 5137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:29:20.318087 5168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.319352 5137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:20.319497 5137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "ad7469d11ca74242a29e9bb97b6d19f2"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "43c5c37812a0f4de8621b09c74d4b79c"
server_key_iv: "c5c4e395f6727c43aeb5f583b56a172a"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.319608 5137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:20.339295 5137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:20.343080 5137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:20.343351 5137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:20.344120 5137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:20.345212 5137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:20.345293 5137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.345382 5137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:20.345429 5137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.355974 5137 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:34539
I20260810 06:29:20.355993 5363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:34539 every 8 connection(s)
I20260810 06:29:20.357059 5137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:29:20.364645 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5137
I20260810 06:29:20.364789 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:29:20.365129 4388 external_mini_cluster.cc:1428] Setting key 69efe952388adef4ac0b9ab65efe9db6
Aug 10 06:29:20 dist-test-slave-45dx krb5kdc[4722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343360, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.2@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:20.372366 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.359047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:52115 (local address 127.4.73.62:45145)
0810 06:29:20.359230 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.359236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.360097 (+ 861us) server_negotiation.cc:407] Connection header received
0810 06:29:20.361524 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.361527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.361580 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.361665 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:20.362939 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.363505 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.364962 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.365214 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.365348 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.368085 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:20.368089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:20.368091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:20.368115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:20.369905 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.370470 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.370477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.370481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.370585 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.370954 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.370960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.370965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.371247 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:20.371380 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.372005 (+ 625us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.372154 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 06:29:20.373476 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.359442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45145 (local address 127.4.73.2:52115)
0810 06:29:20.359930 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.359982 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.361287 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.361858 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.361865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.362243 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:20.362767 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.362781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.363640 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.363647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.364835 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.364849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.365404 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.365409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.365593 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.366301 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:20.366319 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:20.367908 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:20.370043 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:20.370049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:20.370063 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:20.370355 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:20.370690 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:20.370692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:20.370694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:20.370851 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:20.371368 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:20.371375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:20.371716 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.371946 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.372389 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":127,"threads_started":1}
I20260810 06:29:20.374851 5364 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45145
I20260810 06:29:20.375128 5364 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:20.375653 5364 heartbeater.cc:511] Master 127.4.73.62:45145 requested a full tablet report, sending...
I20260810 06:29:20.376847 4797 ts_manager.cc:194] Registered new tserver with Master: ad7469d11ca74242a29e9bb97b6d19f2 (127.4.73.2:34539)
WARNING: no policy specified for kudu/127.4.73.3@KRBTEST.COM; defaulting to no policy
I20260810 06:29:20.377496 4797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.2@KRBTEST.COM'} at 127.4.73.2:52115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
Entry for principal kudu/127.4.73.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab.
I20260810 06:29:20.395311 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45145
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:20.507823 5372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:20.508100 5372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:20.508183 5372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:20.512142 5372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:20.512226 5372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:20.512336 5372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:29:20.517563 5372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.3.keytab
--principal=kudu/127.4.73.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:45145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:20.518567 5372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:20.519680 5372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:20.529428 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:20.529417 5381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:20.529449 5372 server_base.cc:1034] running on GCE node
W20260810 06:29:20.529496 5385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:20.530093 5372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:20.530711 5372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:20.531905 5372 hybrid_clock.cc:648] HybridClock initialized: now 1786343360531891 us; error 24 us; skew 500 ppm
Aug 10 06:29:20 dist-test-slave-45dx krb5kdc[4722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343360, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:20.534593 5372 init.cc:377] Logged in from keytab as kudu/127.4.73.3@KRBTEST.COM (short username kudu)
I20260810 06:29:20.535377 5372 webserver.cc:460] Webserver started at http://127.4.73.3:45505/ using document root <none> and password file <none>
I20260810 06:29:20.535871 5372 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:20.535929 5372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:20.536096 5372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:20.538086 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "88aeab78e5ae4d4ca7b003b1b0a46303"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "fde97c08b311246c14ad523a06d53f2e"
server_key_iv: "2fe6327c4df0e05e832085d152e2ff89"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.538606 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "88aeab78e5ae4d4ca7b003b1b0a46303"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "fde97c08b311246c14ad523a06d53f2e"
server_key_iv: "2fe6327c4df0e05e832085d152e2ff89"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.542276 5372 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:20.544692 5397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.545866 5372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:29:20.545961 5372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "88aeab78e5ae4d4ca7b003b1b0a46303"
format_stamp: "Formatted at 2026-08-10 06:29:20 on dist-test-slave-45dx"
server_key: "fde97c08b311246c14ad523a06d53f2e"
server_key_iv: "2fe6327c4df0e05e832085d152e2ff89"
server_key_version: "encryptionkey@0"
I20260810 06:29:20.546051 5372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:20.582779 5372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:20.585781 5372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:20.585939 5372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:20.586484 5372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:20.587252 5372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:20.587299 5372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.587339 5372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:20.587356 5372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:20.596451 5372 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:42373
I20260810 06:29:20.596488 5595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:42373 every 8 connection(s)
I20260810 06:29:20.597280 5372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:29:20.602367 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5372
I20260810 06:29:20.602519 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:29:20.602881 4388 external_mini_cluster.cc:1428] Setting key d7c35622993b0e463e8778102cff1504
Aug 10 06:29:20 dist-test-slave-45dx krb5kdc[4722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343360, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.3@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:20.611037 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.599045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:50837 (local address 127.4.73.62:45145)
0810 06:29:20.599189 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.599194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.600035 (+ 841us) server_negotiation.cc:407] Connection header received
0810 06:29:20.601410 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.601413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.601460 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.601535 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:20.602847 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.603525 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.604958 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.605209 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.605329 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.607727 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:20.607730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:20.607732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:20.607756 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:20.609265 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.609749 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.609752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.609753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.609826 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.610151 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.610153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.610154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.610319 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:20.610397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.610817 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.610920 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 06:29:20.611805 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.599460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:45145 (local address 127.4.73.3:50837)
0810 06:29:20.599887 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.599929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.601218 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.601657 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.601663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.602003 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:20.602687 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.602711 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.603653 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.603656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.604794 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.604804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.605361 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.605364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.605514 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.606034 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:20.606049 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:20.607580 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:20.609379 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:20.609383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:20.609394 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:20.609663 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:20.609921 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:20.609923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:20.609925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:20.610069 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:20.610408 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:20.610412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:20.610621 (+ 209us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.610770 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.611000 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260810 06:29:20.612975 5596 heartbeater.cc:347] Connected to a master server at 127.4.73.62:45145
I20260810 06:29:20.613211 5596 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:20.613682 5596 heartbeater.cc:511] Master 127.4.73.62:45145 requested a full tablet report, sending...
I20260810 06:29:20.614666 4797 ts_manager.cc:194] Registered new tserver with Master: 88aeab78e5ae4d4ca7b003b1b0a46303 (127.4.73.3:42373)
I20260810 06:29:20.615217 4797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.73.3@KRBTEST.COM'} at 127.4.73.3:50837
I20260810 06:29:20.616803 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:29:20.625741 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.618819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.4.73.62:45145)
0810 06:29:20.618950 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.618954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.619124 (+ 170us) server_negotiation.cc:407] Connection header received
0810 06:29:20.619200 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.619203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.619237 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.619293 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:20.620027 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.620560 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.621366 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.621566 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.621640 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.623012 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:20.623014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:20.623016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:20.623035 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:20.624386 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.624745 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.624747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.624748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.624819 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:20.625050 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:20.625052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:20.625053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:20.625240 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:20.625338 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.625494 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.625575 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 06:29:20.628463 4797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:29:20.631381 4797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:29:20.645593 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.640019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:42373 (local address 127.0.0.1:37200)
0810 06:29:20.641058 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.641073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.641167 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.641687 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.641691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.641711 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:20.641833 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.641839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.642950 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.642953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.643950 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.643960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.645223 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.645226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.645354 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.645374 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.645451 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.645506 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":112,"threads_started":1}
I20260810 06:29:20.645654 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.639628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.4.73.1:45209)
0810 06:29:20.639958 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.639976 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.640001 (+ 25us) server_negotiation.cc:407] Connection header received
0810 06:29:20.640213 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.640220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.640469 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.640643 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:20.640962 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.642153 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.643736 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.644626 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.644725 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.644892 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.645375 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.645493 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":76,"threads_started":1}
I20260810 06:29:20.645946 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.640411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.4.73.3:42373)
0810 06:29:20.640975 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.640987 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.641267 (+ 280us) server_negotiation.cc:407] Connection header received
0810 06:29:20.641347 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.641353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.641541 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.641694 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:20.641973 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.642807 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.644088 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.644964 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.645064 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.645186 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.645675 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.645772 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":154,"threads_started":1}
I20260810 06:29:20.646107 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.640195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:34539 (local address 127.0.0.1:51292)
0810 06:29:20.641204 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.641217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.641302 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.641905 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.641908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.641937 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:20.642011 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.642015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.643405 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.643408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.644503 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.644513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.645704 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.645708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.645791 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.645818 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.645917 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.645993 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":94,"threads_started":1}
I20260810 06:29:20.646539 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.640520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.4.73.2:34539)
0810 06:29:20.640924 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.640937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.641385 (+ 448us) server_negotiation.cc:407] Connection header received
0810 06:29:20.641455 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.641459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.641608 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.641728 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:20.642133 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.643264 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.644636 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.645411 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.645508 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.645635 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.646333 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.646425 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":109,"threads_started":1}
I20260810 06:29:20.648993 5250 tablet_service.cc:1467] Processing CreateTablet for tablet 9072b7e0cd3e4de4a971440bd926fb49 (DEFAULT_TABLE table=test-table [id=342cb9d992334fed81823f89c9ab0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:20.648995 5479 tablet_service.cc:1467] Processing CreateTablet for tablet 9072b7e0cd3e4de4a971440bd926fb49 (DEFAULT_TABLE table=test-table [id=342cb9d992334fed81823f89c9ab0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:20.645593 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.639488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:45209 (local address 127.0.0.1:39224)
0810 06:29:20.639805 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.639854 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.640008 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.640646 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.640652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.640674 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:20.640806 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.640815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.642292 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.642295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.643595 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.643605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.645017 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.645021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.645129 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.645179 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.645320 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.645449 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260810 06:29:20.650097 5250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9072b7e0cd3e4de4a971440bd926fb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:20.650097 5479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9072b7e0cd3e4de4a971440bd926fb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:20.651852 5010 tablet_service.cc:1467] Processing CreateTablet for tablet 9072b7e0cd3e4de4a971440bd926fb49 (DEFAULT_TABLE table=test-table [id=342cb9d992334fed81823f89c9ab0353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:20.652760 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9072b7e0cd3e4de4a971440bd926fb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:20.657349 5623 tablet_bootstrap.cc:492] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Bootstrap starting.
I20260810 06:29:20.657750 5625 tablet_bootstrap.cc:492] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Bootstrap starting.
I20260810 06:29:20.658560 5626 tablet_bootstrap.cc:492] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: Bootstrap starting.
I20260810 06:29:20.659957 5625 tablet_bootstrap.cc:654] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:20.660539 5626 tablet_bootstrap.cc:654] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:20.660712 5623 tablet_bootstrap.cc:654] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:20.661334 5625 log.cc:824] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:20.661356 5626 log.cc:824] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:20.661394 5623 log.cc:824] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:20.663102 5623 tablet_bootstrap.cc:492] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: No bootstrap required, opened a new log
I20260810 06:29:20.663275 5625 tablet_bootstrap.cc:492] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: No bootstrap required, opened a new log
I20260810 06:29:20.663275 5626 tablet_bootstrap.cc:492] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: No bootstrap required, opened a new log
I20260810 06:29:20.663316 5623 ts_tablet_manager.cc:1397] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260810 06:29:20.663462 5625 ts_tablet_manager.cc:1397] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:29:20.663462 5626 ts_tablet_manager.cc:1397] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 06:29:20.666105 5623 raft_consensus.cc:348] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.666407 5623 raft_consensus.cc:374] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:20.666464 5623 raft_consensus.cc:729] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7469d11ca74242a29e9bb97b6d19f2, State: Initialized, Role: FOLLOWER
I20260810 06:29:20.666929 5623 consensus_queue.cc:260] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.667227 5625 raft_consensus.cc:348] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.667505 5625 raft_consensus.cc:374] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:20.667567 5625 raft_consensus.cc:729] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88aeab78e5ae4d4ca7b003b1b0a46303, State: Initialized, Role: FOLLOWER
I20260810 06:29:20.667817 5364 heartbeater.cc:503] Master 127.4.73.62:45145 was elected leader, sending a full tablet report...
I20260810 06:29:20.667675 5626 raft_consensus.cc:348] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.667893 5626 raft_consensus.cc:374] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:20.667938 5623 ts_tablet_manager.cc:1428] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260810 06:29:20.667943 5626 raft_consensus.cc:729] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd69950bf834dcfa29ba71d4c31b58e, State: Initialized, Role: FOLLOWER
I20260810 06:29:20.668016 5625 consensus_queue.cc:260] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.668335 5626 consensus_queue.cc:260] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.668835 5596 heartbeater.cc:503] Master 127.4.73.62:45145 was elected leader, sending a full tablet report...
I20260810 06:29:20.668988 5625 ts_tablet_manager.cc:1428] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 06:29:20.669178 5626 ts_tablet_manager.cc:1428] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 06:29:20.669559 5130 heartbeater.cc:503] Master 127.4.73.62:45145 was elected leader, sending a full tablet report...
I20260810 06:29:20.836724 5634 raft_consensus.cc:482] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:20.836973 5634 raft_consensus.cc:504] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.838207 5634 leader_election.cc:288] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cd69950bf834dcfa29ba71d4c31b58e (127.4.73.1:45209), 88aeab78e5ae4d4ca7b003b1b0a46303 (127.4.73.3:42373)
I20260810 06:29:20.841887 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.838551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:52609 (local address 127.4.73.1:45209)
0810 06:29:20.838681 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.838686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.838730 (+ 44us) server_negotiation.cc:407] Connection header received
0810 06:29:20.838792 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.838795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.838851 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.838929 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:20.839231 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.839737 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.840848 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.841297 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.841423 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.841482 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.841689 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.841766 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 06:29:20.841887 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.838434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:45209 (local address 127.4.73.2:52609)
0810 06:29:20.838548 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.838569 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.838666 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.838959 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.838963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.838982 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:20.839102 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.839108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.839862 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.839865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.840718 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.840728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.841425 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.841429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.841562 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.841580 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.841672 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.841754 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 06:29:20.842077 5637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.838474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:42373 (local address 127.4.73.2:60833)
0810 06:29:20.838773 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.838784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:20.838889 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:20.839199 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:20.839202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:20.839218 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:20.839338 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.839344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.840065 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.840069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.840932 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:20.840948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.841620 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.841624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:20.841751 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.841770 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 06:29:20.841864 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 06:29:20.841935 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260810 06:29:20.842067 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.838817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:60833 (local address 127.4.73.3:42373)
0810 06:29:20.838985 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.838990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.839011 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:29:20.839056 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.839058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.839095 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.839185 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:20.839459 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.839947 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.841071 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.841475 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.841615 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.841677 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.841890 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.841960 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 06:29:20.842617 5044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9072b7e0cd3e4de4a971440bd926fb49" candidate_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" is_pre_election: true
I20260810 06:29:20.842619 5511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9072b7e0cd3e4de4a971440bd926fb49" candidate_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" is_pre_election: true
I20260810 06:29:20.842944 5511 raft_consensus.cc:2454] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7469d11ca74242a29e9bb97b6d19f2 in term 0.
I20260810 06:29:20.842935 5044 raft_consensus.cc:2454] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7469d11ca74242a29e9bb97b6d19f2 in term 0.
I20260810 06:29:20.843461 5174 leader_election.cc:302] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88aeab78e5ae4d4ca7b003b1b0a46303, ad7469d11ca74242a29e9bb97b6d19f2; no voters:
I20260810 06:29:20.843756 5634 raft_consensus.cc:2781] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:29:20.843845 5634 raft_consensus.cc:482] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:29:20.843899 5634 raft_consensus.cc:3037] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:20.844982 5634 raft_consensus.cc:504] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
I20260810 06:29:20.845306 5634 leader_election.cc:288] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8cd69950bf834dcfa29ba71d4c31b58e (127.4.73.1:45209), 88aeab78e5ae4d4ca7b003b1b0a46303 (127.4.73.3:42373)
I20260810 06:29:20.845613 5044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9072b7e0cd3e4de4a971440bd926fb49" candidate_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd69950bf834dcfa29ba71d4c31b58e"
I20260810 06:29:20.845686 5511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9072b7e0cd3e4de4a971440bd926fb49" candidate_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303"
I20260810 06:29:20.845752 5044 raft_consensus.cc:3037] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:20.845835 5511 raft_consensus.cc:3037] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:20.846856 5511 raft_consensus.cc:2454] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7469d11ca74242a29e9bb97b6d19f2 in term 1.
I20260810 06:29:20.846872 5044 raft_consensus.cc:2454] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7469d11ca74242a29e9bb97b6d19f2 in term 1.
I20260810 06:29:20.847193 5174 leader_election.cc:302] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88aeab78e5ae4d4ca7b003b1b0a46303, ad7469d11ca74242a29e9bb97b6d19f2; no voters:
I20260810 06:29:20.847380 5634 raft_consensus.cc:2781] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:20.847671 5634 raft_consensus.cc:686] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 1 LEADER]: Becoming Leader. State: Replica: ad7469d11ca74242a29e9bb97b6d19f2, State: Running, Role: LEADER
I20260810 06:29:20.848021 5634 consensus_queue.cc:237] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } }
W20260810 06:29:20.848160 5597 tablet.cc:2369] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:20.851416 4796 catalog_manager.cc:5495] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad7469d11ca74242a29e9bb97b6d19f2 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad7469d11ca74242a29e9bb97b6d19f2" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 34539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 } health_report { overall_health: UNKNOWN } } }
W20260810 06:29:20.858242 5365 tablet.cc:2369] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:20.885547 5622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.881940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.4.73.2:34539)
0810 06:29:20.882079 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:20.882083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:20.882522 (+ 439us) server_negotiation.cc:407] Connection header received
0810 06:29:20.882645 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:20.882648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:20.882704 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:20.882810 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:20.883051 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.883530 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.884319 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:20.884522 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:20.884598 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:20.884867 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:20.885232 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:20.885318 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:20.885379 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 06:29:20.885426 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 06:29:20.887722 5131 tablet.cc:2369] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:20.891566 5044 raft_consensus.cc:1261] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 1 FOLLOWER]: Refusing update from remote peer ad7469d11ca74242a29e9bb97b6d19f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:20.891566 5511 raft_consensus.cc:1261] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 1 FOLLOWER]: Refusing update from remote peer ad7469d11ca74242a29e9bb97b6d19f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:20.892411 5634 consensus_queue.cc:1035] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd69950bf834dcfa29ba71d4c31b58e" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:20.892634 5639 consensus_queue.cc:1035] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88aeab78e5ae4d4ca7b003b1b0a46303" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:20.898923 5645 mvcc.cc:205] Tried to move back new op lower bound from 7316862406204432384 to 7316862406036733952. Current Snapshot: MvccSnapshot[applied={T|T < 7316862406204432384}]
I20260810 06:29:20.899912 5646 mvcc.cc:205] Tried to move back new op lower bound from 7316862406204432384 to 7316862406036733952. Current Snapshot: MvccSnapshot[applied={T|T < 7316862406204432384}]
I20260810 06:29:20.909361 4796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:20.909557 4796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:20.912009 4796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bd2a27409944751bffd286a0e9c26e6: Sending DeleteTablet for 3 replicas of tablet 9072b7e0cd3e4de4a971440bd926fb49
I20260810 06:29:20.912881 5479 tablet_service.cc:1514] Processing DeleteTablet for tablet 9072b7e0cd3e4de4a971440bd926fb49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:20 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:37200
I20260810 06:29:20.912892 5010 tablet_service.cc:1514] Processing DeleteTablet for tablet 9072b7e0cd3e4de4a971440bd926fb49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:20 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:39224
I20260810 06:29:20.913393 5662 tablet_replica.cc:330] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: stopping tablet replica
I20260810 06:29:20.913565 5250 tablet_service.cc:1514] Processing DeleteTablet for tablet 9072b7e0cd3e4de4a971440bd926fb49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:20 UTC) from {username='kudu', principal='kudu/127.4.73.62@KRBTEST.COM'} at 127.0.0.1:51292
I20260810 06:29:20.913689 5663 tablet_replica.cc:330] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e: stopping tablet replica
I20260810 06:29:20.914007 5664 tablet_replica.cc:330] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: stopping tablet replica
I20260810 06:29:20.914017 5663 raft_consensus.cc:2229] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:20.914225 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 4912
I20260810 06:29:20.914354 5663 raft_consensus.cc:2258] T 9072b7e0cd3e4de4a971440bd926fb49 P 8cd69950bf834dcfa29ba71d4c31b58e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:20.943866 5662 raft_consensus.cc:2229] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:20.944330 5662 raft_consensus.cc:2258] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:20.947117 5662 ts_tablet_manager.cc:1905] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:20.950011 5662 ts_tablet_manager.cc:1918] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:20.950107 5662 log.cc:1196] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/9072b7e0cd3e4de4a971440bd926fb49
I20260810 06:29:20.950470 5662 ts_tablet_manager.cc:1939] T 9072b7e0cd3e4de4a971440bd926fb49 P 88aeab78e5ae4d4ca7b003b1b0a46303: Deleting consensus metadata
I20260810 06:29:20.914294 5664 raft_consensus.cc:2229] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:29:20.952426 5664 raft_consensus.cc:2258] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:20.953094 4774 catalog_manager.cc:4841] TS 88aeab78e5ae4d4ca7b003b1b0a46303 (127.4.73.3:42373): tablet 9072b7e0cd3e4de4a971440bd926fb49 (table test-table [id=342cb9d992334fed81823f89c9ab0353]) successfully deleted
I20260810 06:29:20.954241 5664 ts_tablet_manager.cc:1905] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:20.956506 5664 ts_tablet_manager.cc:1918] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:20.956625 5664 log.cc:1196] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/9072b7e0cd3e4de4a971440bd926fb49
I20260810 06:29:20.956905 5664 ts_tablet_manager.cc:1939] T 9072b7e0cd3e4de4a971440bd926fb49 P ad7469d11ca74242a29e9bb97b6d19f2: Deleting consensus metadata
I20260810 06:29:20.957995 4770 catalog_manager.cc:4841] TS ad7469d11ca74242a29e9bb97b6d19f2 (127.4.73.2:34539): tablet 9072b7e0cd3e4de4a971440bd926fb49 (table test-table [id=342cb9d992334fed81823f89c9ab0353]) successfully deleted
W20260810 06:29:20.961553 4770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:45209 (error 108)
I20260810 06:29:20.961683 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5137
I20260810 06:29:20.965166 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:20.964828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:45209 (local address 127.0.0.1:39240)
0810 06:29:20.964976 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:20.965072 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 06:29:20.965375 4770 catalog_manager.cc:4584] TS 8cd69950bf834dcfa29ba71d4c31b58e (127.4.73.1:45209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9072b7e0cd3e4de4a971440bd926fb49: Network error: Client connection negotiation failed: client connection to 127.4.73.1:45209: connect: Connection refused (error 111)
I20260810 06:29:20.969931 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5372
I20260810 06:29:20.980165 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 4741
2026-08-10T06:29:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2665 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T06:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:21Z Disabled control of system clock
I20260810 06:29:21.013616 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38907
--webserver_interface=127.4.73.62
--webserver_port=0
--builtin_ntp_servers=127.4.73.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:38907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:29:21.123456 5672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:21.123708 5672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:21.123741 5672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:21.127744 5672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:21.127801 5672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:21.127816 5672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:21.127832 5672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:21.127847 5672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:21.132675 5672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:38907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:38907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:21.133755 5672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:21.135143 5672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:21.143672 5680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.143675 5681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.143682 5687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:21.143719 5672 server_base.cc:1034] running on GCE node
I20260810 06:29:21.144488 5672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:21.145525 5672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:21.146715 5672 hybrid_clock.cc:648] HybridClock initialized: now 1786343361146701 us; error 35 us; skew 500 ppm
I20260810 06:29:21.148708 5672 webserver.cc:460] Webserver started at http://127.4.73.62:43107/ using document root <none> and password file <none>
I20260810 06:29:21.149257 5672 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:21.149350 5672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:21.149571 5672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:21.151489 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "c8267b78ca584cf894c154b0b2ffe093"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "d4eff7999e4f54409f1164397a425a32"
server_key_iv: "3422e9e4ad6a3211f4751a3f8ae0b471"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.152060 5672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "c8267b78ca584cf894c154b0b2ffe093"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "d4eff7999e4f54409f1164397a425a32"
server_key_iv: "3422e9e4ad6a3211f4751a3f8ae0b471"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.155478 5672 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 06:29:21.157871 5698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.159029 5672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:29:21.159155 5672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "c8267b78ca584cf894c154b0b2ffe093"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "d4eff7999e4f54409f1164397a425a32"
server_key_iv: "3422e9e4ad6a3211f4751a3f8ae0b471"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.159255 5672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:21.174399 5672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:21.175040 5672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:21.175226 5672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:21.182204 5672 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:38907
I20260810 06:29:21.182251 5780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:38907 every 8 connection(s)
I20260810 06:29:21.183123 5672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:21.186126 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:21.189319 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5672
I20260810 06:29:21.189461 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:21.189725 4388 external_mini_cluster.cc:1428] Setting key fec5ddb3b4657e6ab53b4e1350687018
I20260810 06:29:21.191910 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093: Bootstrap starting.
I20260810 06:29:21.194142 5781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:21.194887 5781 log.cc:824] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:21.196782 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093: No bootstrap required, opened a new log
I20260810 06:29:21.199196 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.191084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.4.73.62:38907)
0810 06:29:21.191703 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.191717 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.191751 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:29:21.192480 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.192499 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.192884 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.193206 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:21.193642 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.194632 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.195569 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.196043 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.196216 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.197137 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:21.197145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:21.197162 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:21.197665 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:21.197735 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:21.197762 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:21.197874 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.198012 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.198302 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":143,"threads_started":1}
I20260810 06:29:21.201216 5781 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8267b78ca584cf894c154b0b2ffe093" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38907 } }
I20260810 06:29:21.201486 5781 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:21.201552 5781 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8267b78ca584cf894c154b0b2ffe093, State: Initialized, Role: FOLLOWER
I20260810 06:29:21.202015 5781 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8267b78ca584cf894c154b0b2ffe093" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38907 } }
I20260810 06:29:21.202165 5781 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:21.202230 5781 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:21.202351 5781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:21.203205 5781 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8267b78ca584cf894c154b0b2ffe093" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38907 } }
I20260810 06:29:21.203564 5781 leader_election.cc:302] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8267b78ca584cf894c154b0b2ffe093; no voters:
I20260810 06:29:21.203868 5781 leader_election.cc:288] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:21.204002 5787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:21.204234 5787 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [term 1 LEADER]: Becoming Leader. State: Replica: c8267b78ca584cf894c154b0b2ffe093, State: Running, Role: LEADER
I20260810 06:29:21.204583 5787 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8267b78ca584cf894c154b0b2ffe093" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38907 } }
I20260810 06:29:21.205195 5781 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:21.206385 5788 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8267b78ca584cf894c154b0b2ffe093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8267b78ca584cf894c154b0b2ffe093" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38907 } } }
I20260810 06:29:21.206410 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8267b78ca584cf894c154b0b2ffe093. Latest consensus state: current_term: 1 leader_uuid: "c8267b78ca584cf894c154b0b2ffe093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8267b78ca584cf894c154b0b2ffe093" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 38907 } } }
I20260810 06:29:21.206545 5788 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:21.206547 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:21.206921 5802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:21.209956 5802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:21.215468 5802 catalog_manager.cc:1347] Generated new cluster ID: 79fcd58f17a448dba9a7e01390f1cf2e
I20260810 06:29:21.215538 5802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:21.232968 5802 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:21.233913 5802 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:21.241621 5802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093: Generated new TSK 0
I20260810 06:29:21.242308 5802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:21.245167 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.1:0
--local_ip_for_outbound_sockets=127.4.73.1
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38907
--builtin_ntp_servers=127.4.73.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 06:29:21.354527 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:21.354753 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:21.354787 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:21.358530 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:21.358577 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:21.358628 5819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.1
I20260810 06:29:21.363071 5819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.73.1
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.1
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:21.363979 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:21.365301 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:21.374493 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.374501 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.374514 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:21.374724 5819 server_base.cc:1034] running on GCE node
I20260810 06:29:21.375212 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:21.375828 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:21.376991 5819 hybrid_clock.cc:648] HybridClock initialized: now 1786343361376978 us; error 24 us; skew 500 ppm
I20260810 06:29:21.378901 5819 webserver.cc:460] Webserver started at http://127.4.73.1:42491/ using document root <none> and password file <none>
I20260810 06:29:21.379459 5819 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:21.379519 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:21.379746 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:21.381781 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data/instance:
uuid: "0dadee1fe26e4ddc899c012bddf0bf3a"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "0662e80974693193577dcde37f53dd23"
server_key_iv: "30974e9ef76ebff2ed0d62b3e9be1afa"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.382369 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance:
uuid: "0dadee1fe26e4ddc899c012bddf0bf3a"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "0662e80974693193577dcde37f53dd23"
server_key_iv: "30974e9ef76ebff2ed0d62b3e9be1afa"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.385799 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:21.388113 5842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.389372 5819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:21.389510 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/wal
uuid: "0dadee1fe26e4ddc899c012bddf0bf3a"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "0662e80974693193577dcde37f53dd23"
server_key_iv: "30974e9ef76ebff2ed0d62b3e9be1afa"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.389621 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:21.416576 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:21.417296 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:21.417510 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:21.418154 5819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:21.419117 5819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:21.419190 5819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.419260 5819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:21.419310 5819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.428200 5819 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.1:38461
I20260810 06:29:21.428236 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.1:38461 every 8 connection(s)
I20260810 06:29:21.429113 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/data/info.pb
I20260810 06:29:21.432287 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 5819
I20260810 06:29:21.432444 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-0/wal/instance
I20260810 06:29:21.432749 4388 external_mini_cluster.cc:1428] Setting key 2c48c2235e431bb97d57e7c95579f709
I20260810 06:29:21.434950 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.2:0
--local_ip_for_outbound_sockets=127.4.73.2
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38907
--builtin_ntp_servers=127.4.73.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:29:21.439385 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.430851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.1:47277 (local address 127.4.73.62:38907)
0810 06:29:21.431006 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.431011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.432053 (+ 1042us) server_negotiation.cc:407] Connection header received
0810 06:29:21.433883 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.433887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.433933 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.434031 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:21.435236 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.435749 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.437217 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.437636 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.437770 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.438888 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:21.438891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:21.438894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:21.438906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:21.438929 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:21.438944 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:21.439023 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.439181 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.439240 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 06:29:21.440207 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.431280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38907 (local address 127.4.73.1:47277)
0810 06:29:21.431903 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.431943 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.433672 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.434235 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.434243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.434351 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:21.435032 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.435054 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.435905 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.435911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.437097 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.437107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.437796 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.437799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.437964 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.438610 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:21.438612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:21.438651 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:21.438711 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:21.439025 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:21.439037 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:21.439044 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.439168 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.439426 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260810 06:29:21.441547 6012 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38907
I20260810 06:29:21.441771 6012 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:21.442265 6012 heartbeater.cc:511] Master 127.4.73.62:38907 requested a full tablet report, sending...
I20260810 06:29:21.443979 5727 ts_manager.cc:194] Registered new tserver with Master: 0dadee1fe26e4ddc899c012bddf0bf3a (127.4.73.1:38461)
I20260810 06:29:21.445667 5727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.1:47277
W20260810 06:29:21.543578 6017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:21.543789 6017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:21.543824 6017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:21.547595 6017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:21.547645 6017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:21.547720 6017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.2
I20260810 06:29:21.552299 6017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.73.2
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.2
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:21.553256 6017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:21.554491 6017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:21.563632 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.563681 6028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.563699 6029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:21.563851 6017 server_base.cc:1034] running on GCE node
I20260810 06:29:21.564472 6017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:21.565033 6017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:21.566236 6017 hybrid_clock.cc:648] HybridClock initialized: now 1786343361566206 us; error 44 us; skew 500 ppm
I20260810 06:29:21.567946 6017 webserver.cc:460] Webserver started at http://127.4.73.2:34127/ using document root <none> and password file <none>
I20260810 06:29:21.568562 6017 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:21.568655 6017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:21.568897 6017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:21.570888 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data/instance:
uuid: "5f6d530faceb42e68529b01bbe675851"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "2b5887221ecfdefec680d5c84a3ae556"
server_key_iv: "17e669b03fe329bf60f1d5343b9b358f"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.571533 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance:
uuid: "5f6d530faceb42e68529b01bbe675851"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "2b5887221ecfdefec680d5c84a3ae556"
server_key_iv: "17e669b03fe329bf60f1d5343b9b358f"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.574987 6017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:29:21.577316 6042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.578480 6017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:21.578625 6017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal
uuid: "5f6d530faceb42e68529b01bbe675851"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "2b5887221ecfdefec680d5c84a3ae556"
server_key_iv: "17e669b03fe329bf60f1d5343b9b358f"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.578742 6017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:21.594866 6017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:21.595494 6017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:21.595677 6017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:21.596225 6017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:21.597141 6017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:21.597211 6017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.597285 6017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:21.597326 6017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.606288 6017 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.2:35951
I20260810 06:29:21.606343 6221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.2:35951 every 8 connection(s)
I20260810 06:29:21.607291 6017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/data/info.pb
I20260810 06:29:21.612293 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6017
I20260810 06:29:21.612468 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal/instance
I20260810 06:29:21.612777 4388 external_mini_cluster.cc:1428] Setting key 0172ad0834e5f4d4ecaaffe26010cf7c
I20260810 06:29:21.614969 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.73.3:0
--local_ip_for_outbound_sockets=127.4.73.3
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38907
--builtin_ntp_servers=127.4.73.20:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 06:29:21.617532 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.609074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:46083 (local address 127.4.73.62:38907)
0810 06:29:21.609238 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.609242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.610206 (+ 964us) server_negotiation.cc:407] Connection header received
0810 06:29:21.611609 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.611613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.611672 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.611779 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:21.612956 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.613660 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.615314 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.615733 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.615855 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.617007 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:21.617015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:21.617017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:21.617029 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:21.617049 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:21.617057 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:21.617134 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.617324 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.617389 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 06:29:21.618353 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.609444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38907 (local address 127.4.73.2:46083)
0810 06:29:21.610061 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.610100 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.611401 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.611952 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.611958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.612064 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:21.612768 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.612789 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.613798 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.613802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.615082 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.615164 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.615904 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.615907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.616042 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.616715 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:21.616716 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:21.616755 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:21.616816 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:21.617150 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:21.617162 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:21.617172 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.617323 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.617608 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260810 06:29:21.619494 6222 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38907
I20260810 06:29:21.619740 6222 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:21.620229 6222 heartbeater.cc:511] Master 127.4.73.62:38907 requested a full tablet report, sending...
I20260810 06:29:21.621467 5727 ts_manager.cc:194] Registered new tserver with Master: 5f6d530faceb42e68529b01bbe675851 (127.4.73.2:35951)
I20260810 06:29:21.622349 5727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.2:46083
W20260810 06:29:21.731866 6228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:21.732187 6228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:21.732273 6228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:21.736927 6228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:21.737004 6228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:21.737094 6228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.73.3
I20260810 06:29:21.742158 6228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.73.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.73.3
--webserver_port=0
--tserver_master_addrs=127.4.73.62:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.73.3
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:21.743438 6228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:21.744912 6228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:21.754508 6242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.754519 6239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:21.754508 6237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:21.754740 6228 server_base.cc:1034] running on GCE node
I20260810 06:29:21.755185 6228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:21.755822 6228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:21.756989 6228 hybrid_clock.cc:648] HybridClock initialized: now 1786343361756976 us; error 25 us; skew 500 ppm
I20260810 06:29:21.758848 6228 webserver.cc:460] Webserver started at http://127.4.73.3:35311/ using document root <none> and password file <none>
I20260810 06:29:21.759413 6228 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:21.759477 6228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:21.759735 6228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:21.761823 6228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data/instance:
uuid: "0bc542c1dbe54ccfbee328c2d723c0d0"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "7a5a63f1b1122dff807e92b78dca281b"
server_key_iv: "29dd3dcff65cb5c2be2870aa4fccb1e6"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.762401 6228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance:
uuid: "0bc542c1dbe54ccfbee328c2d723c0d0"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "7a5a63f1b1122dff807e92b78dca281b"
server_key_iv: "29dd3dcff65cb5c2be2870aa4fccb1e6"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.766144 6228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:21.768642 6256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.769878 6228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:29:21.770007 6228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal
uuid: "0bc542c1dbe54ccfbee328c2d723c0d0"
format_stamp: "Formatted at 2026-08-10 06:29:21 on dist-test-slave-45dx"
server_key: "7a5a63f1b1122dff807e92b78dca281b"
server_key_iv: "29dd3dcff65cb5c2be2870aa4fccb1e6"
server_key_version: "encryptionkey@0"
I20260810 06:29:21.770125 6228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:21.790906 6228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:21.791643 6228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:21.791863 6228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:21.792577 6228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 06:29:21.793653 6228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 06:29:21.793710 6228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.793783 6228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 06:29:21.793823 6228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:21.802711 6228 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.3:37847
I20260810 06:29:21.802739 6454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.3:37847 every 8 connection(s)
I20260810 06:29:21.803591 6228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/data/info.pb
I20260810 06:29:21.812872 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.805537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.3:36721 (local address 127.4.73.62:38907)
0810 06:29:21.805733 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.805737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.806866 (+ 1129us) server_negotiation.cc:407] Connection header received
0810 06:29:21.808326 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.808330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.808385 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.808469 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:21.809338 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.809861 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.810965 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.811183 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.811264 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.812461 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:21.812464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:21.812466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:21.812477 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:21.812495 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:21.812502 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:21.812559 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.812705 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.812754 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 06:29:21.813573 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6228
I20260810 06:29:21.813716 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal/instance
I20260810 06:29:21.813731 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.805961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:38907 (local address 127.4.73.3:36721)
0810 06:29:21.806713 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.806764 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.808109 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.808637 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.808644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.808727 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:21.809204 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.809218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.809969 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.809971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.810844 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.810853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.811312 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.811315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.811473 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.812159 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 06:29:21.812161 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:21.812202 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 06:29:21.812264 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:21.812577 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:21.812584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:21.812588 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.812683 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.812896 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":147,"threads_started":1}
I20260810 06:29:21.814098 4388 external_mini_cluster.cc:1428] Setting key 507049db9b3807d5aa54b89da7e00231
I20260810 06:29:21.815165 6455 heartbeater.cc:347] Connected to a master server at 127.4.73.62:38907
I20260810 06:29:21.815405 6455 heartbeater.cc:464] Registering TS with master...
I20260810 06:29:21.816052 6455 heartbeater.cc:511] Master 127.4.73.62:38907 requested a full tablet report, sending...
I20260810 06:29:21.817121 5727 ts_manager.cc:194] Registered new tserver with Master: 0bc542c1dbe54ccfbee328c2d723c0d0 (127.4.73.3:37847)
I20260810 06:29:21.817699 5727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.73.3:36721
I20260810 06:29:21.828377 4388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 06:29:21.834323 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.830354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.4.73.62:38907)
0810 06:29:21.830536 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.830540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.830705 (+ 165us) server_negotiation.cc:407] Connection header received
0810 06:29:21.830862 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.830865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.830913 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.830993 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:21.831305 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.831802 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.832752 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.832974 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.833085 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.833865 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:21.833869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:21.833871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:21.833901 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:21.833919 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:21.833927 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:21.834014 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.834133 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.834188 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 06:29:21.836922 5727 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 06:29:21.839862 5727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 06:29:21.854099 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.848123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.2:35951 (local address 127.0.0.1:53590)
0810 06:29:21.848583 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.848637 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.848766 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.849514 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.849521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.849544 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:21.849664 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.849669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.851049 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.851053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.852445 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.852456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.853535 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.853538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.853661 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.853712 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.853821 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.853941 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":77,"thread_start_us":94,"threads_started":1}
I20260810 06:29:21.854116 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.848339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.4.73.2:35951)
0810 06:29:21.848676 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.848697 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.848724 (+ 27us) server_negotiation.cc:407] Connection header received
0810 06:29:21.848947 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.848957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.849190 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.849377 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:21.849785 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.850924 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.852606 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.853404 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.853523 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.853712 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.853848 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.853949 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":110,"threads_started":1}
I20260810 06:29:21.854314 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.848877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:37847 (local address 127.0.0.1:43186)
0810 06:29:21.849662 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.849676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.849774 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.850338 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.850341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.850359 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:21.850460 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.850468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.851754 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.851757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.852901 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.852910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.853944 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.853947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.854032 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.854051 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.854121 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.854191 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":94,"thread_start_us":94,"threads_started":1}
I20260810 06:29:21.854571 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.849198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:38461 (local address 127.0.0.1:58998)
0810 06:29:21.850036 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.850048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.850139 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.850527 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.850532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.850554 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:21.850644 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.850649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.851737 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.851740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.852938 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.852949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.854174 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.854178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.854272 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.854301 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.854381 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.854447 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":25,"thread_start_us":87,"threads_started":1}
I20260810 06:29:21.854756 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.849025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.4.73.3:37847)
0810 06:29:21.849478 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.849492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.849910 (+ 418us) server_negotiation.cc:407] Connection header received
0810 06:29:21.849999 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.850004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.850194 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.850359 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:21.850588 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.851623 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.853025 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.853597 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.853696 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.853882 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.854476 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.854582 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":120,"threads_started":1}
I20260810 06:29:21.854836 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.849622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.4.73.1:38461)
0810 06:29:21.849995 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.850006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.850140 (+ 134us) server_negotiation.cc:407] Connection header received
0810 06:29:21.850240 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.850245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.850405 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.850550 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:21.850731 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.851617 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.853088 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.853835 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.853931 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.854103 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.854415 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.854511 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":150,"threads_started":1}
I20260810 06:29:21.857738 6343 tablet_service.cc:1467] Processing CreateTablet for tablet 804962bf4dd14ae6b70049ae370a9477 (DEFAULT_TABLE table=test-table [id=e4118aa59b2446d6b6a0466a71ae7ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:21.857930 5918 tablet_service.cc:1467] Processing CreateTablet for tablet 804962bf4dd14ae6b70049ae370a9477 (DEFAULT_TABLE table=test-table [id=e4118aa59b2446d6b6a0466a71ae7ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:21.858961 5918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804962bf4dd14ae6b70049ae370a9477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:21.859174 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804962bf4dd14ae6b70049ae370a9477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:21.859994 6120 tablet_service.cc:1467] Processing CreateTablet for tablet 804962bf4dd14ae6b70049ae370a9477 (DEFAULT_TABLE table=test-table [id=e4118aa59b2446d6b6a0466a71ae7ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 06:29:21.860863 6120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 804962bf4dd14ae6b70049ae370a9477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:21.866501 6493 tablet_bootstrap.cc:492] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a: Bootstrap starting.
I20260810 06:29:21.867064 6496 tablet_bootstrap.cc:492] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Bootstrap starting.
I20260810 06:29:21.867275 6498 tablet_bootstrap.cc:492] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Bootstrap starting.
I20260810 06:29:21.869065 6493 tablet_bootstrap.cc:654] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:21.869270 6498 tablet_bootstrap.cc:654] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:21.869270 6496 tablet_bootstrap.cc:654] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:21.870057 6493 log.cc:824] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:21.870116 6496 log.cc:824] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:21.870116 6498 log.cc:824] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:21.871855 6496 tablet_bootstrap.cc:492] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: No bootstrap required, opened a new log
I20260810 06:29:21.871999 6496 ts_tablet_manager.cc:1397] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:29:21.872040 6493 tablet_bootstrap.cc:492] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a: No bootstrap required, opened a new log
I20260810 06:29:21.872040 6498 tablet_bootstrap.cc:492] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: No bootstrap required, opened a new log
I20260810 06:29:21.872205 6493 ts_tablet_manager.cc:1397] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 06:29:21.872205 6498 ts_tablet_manager.cc:1397] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 06:29:21.875432 6496 raft_consensus.cc:348] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.875675 6496 raft_consensus.cc:374] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:21.875730 6496 raft_consensus.cc:729] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc542c1dbe54ccfbee328c2d723c0d0, State: Initialized, Role: FOLLOWER
I20260810 06:29:21.876187 6496 consensus_queue.cc:260] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.876444 6493 raft_consensus.cc:348] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.876682 6493 raft_consensus.cc:374] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:21.876746 6493 raft_consensus.cc:729] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dadee1fe26e4ddc899c012bddf0bf3a, State: Initialized, Role: FOLLOWER
I20260810 06:29:21.876595 6498 raft_consensus.cc:348] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.876828 6498 raft_consensus.cc:374] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:21.876895 6498 raft_consensus.cc:729] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6d530faceb42e68529b01bbe675851, State: Initialized, Role: FOLLOWER
I20260810 06:29:21.877272 6496 ts_tablet_manager.cc:1428] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260810 06:29:21.877208 6493 consensus_queue.cc:260] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.877238 6498 consensus_queue.cc:260] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.877674 6455 heartbeater.cc:503] Master 127.4.73.62:38907 was elected leader, sending a full tablet report...
I20260810 06:29:21.878127 6498 ts_tablet_manager.cc:1428] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260810 06:29:21.878100 6222 heartbeater.cc:503] Master 127.4.73.62:38907 was elected leader, sending a full tablet report...
I20260810 06:29:21.878648 6012 heartbeater.cc:503] Master 127.4.73.62:38907 was elected leader, sending a full tablet report...
I20260810 06:29:21.880023 6493 ts_tablet_manager.cc:1428] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260810 06:29:21.930218 6014 tablet.cc:2369] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 06:29:21.941378 6505 raft_consensus.cc:482] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 06:29:21.941614 6505 raft_consensus.cc:504] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.943017 6505 leader_election.cc:288] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc542c1dbe54ccfbee328c2d723c0d0 (127.4.73.3:37847), 0dadee1fe26e4ddc899c012bddf0bf3a (127.4.73.1:38461)
I20260810 06:29:21.947090 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.943197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.3:37847 (local address 127.4.73.2:36705)
0810 06:29:21.943306 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.943324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.943441 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.943704 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.943707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.943730 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:21.943865 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.943873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.944891 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.944893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.945981 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.945992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.946719 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.946722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.946799 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.946820 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.946885 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.946942 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 06:29:21.947137 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.943321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:36705 (local address 127.4.73.3:37847)
0810 06:29:21.943476 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.943479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.943492 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:21.943544 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.943546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.943584 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.943687 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:21.943992 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.944777 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.946114 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.946606 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.946711 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.946773 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.946920 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.947012 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 06:29:21.947557 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.943719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.73.2:47559 (local address 127.4.73.1:38461)
0810 06:29:21.943845 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.943851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.944044 (+ 193us) server_negotiation.cc:407] Connection header received
0810 06:29:21.944126 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.944129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.944177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.944262 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 06:29:21.944560 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.945159 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.946375 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.946941 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.947040 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.947094 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.947350 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.947416 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:29:21.947829 6380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804962bf4dd14ae6b70049ae370a9477" candidate_uuid: "5f6d530faceb42e68529b01bbe675851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" is_pre_election: true
I20260810 06:29:21.947961 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.943474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:38461 (local address 127.4.73.2:47559)
0810 06:29:21.943946 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:21.943962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:21.944056 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:21.944276 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:21.944278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:21.944291 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 06:29:21.944447 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.944452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.945271 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.945275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.946270 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:21.946277 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.947076 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.947079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:21.947180 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.947205 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 06:29:21.947731 (+ 526us) client_negotiation.cc:240] Negotiation successful
0810 06:29:21.947808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":146,"threads_started":1}
I20260810 06:29:21.948174 6380 raft_consensus.cc:2454] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f6d530faceb42e68529b01bbe675851 in term 0.
I20260810 06:29:21.948544 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804962bf4dd14ae6b70049ae370a9477" candidate_uuid: "5f6d530faceb42e68529b01bbe675851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" is_pre_election: true
I20260810 06:29:21.948797 6046 leader_election.cc:302] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc542c1dbe54ccfbee328c2d723c0d0, 5f6d530faceb42e68529b01bbe675851; no voters:
I20260810 06:29:21.948839 5946 raft_consensus.cc:2454] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f6d530faceb42e68529b01bbe675851 in term 0.
I20260810 06:29:21.949064 6505 raft_consensus.cc:2781] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 06:29:21.949187 6505 raft_consensus.cc:482] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 06:29:21.949237 6505 raft_consensus.cc:3037] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:21.950313 6505 raft_consensus.cc:504] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.950765 6505 leader_election.cc:288] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc542c1dbe54ccfbee328c2d723c0d0 (127.4.73.3:37847), 0dadee1fe26e4ddc899c012bddf0bf3a (127.4.73.1:38461)
I20260810 06:29:21.951220 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804962bf4dd14ae6b70049ae370a9477" candidate_uuid: "5f6d530faceb42e68529b01bbe675851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a"
I20260810 06:29:21.951218 6380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "804962bf4dd14ae6b70049ae370a9477" candidate_uuid: "5f6d530faceb42e68529b01bbe675851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0"
I20260810 06:29:21.951323 5946 raft_consensus.cc:3037] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:21.951362 6380 raft_consensus.cc:3037] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:21.952425 6380 raft_consensus.cc:2454] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f6d530faceb42e68529b01bbe675851 in term 1.
I20260810 06:29:21.952462 5946 raft_consensus.cc:2454] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f6d530faceb42e68529b01bbe675851 in term 1.
I20260810 06:29:21.952720 6046 leader_election.cc:302] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc542c1dbe54ccfbee328c2d723c0d0, 5f6d530faceb42e68529b01bbe675851; no voters:
I20260810 06:29:21.952944 6505 raft_consensus.cc:2781] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:21.953162 6505 raft_consensus.cc:686] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 1 LEADER]: Becoming Leader. State: Replica: 5f6d530faceb42e68529b01bbe675851, State: Running, Role: LEADER
I20260810 06:29:21.953511 6505 consensus_queue.cc:237] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } }
I20260810 06:29:21.956705 5727 catalog_manager.cc:5495] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f6d530faceb42e68529b01bbe675851 (127.4.73.2). New cstate: current_term: 1 leader_uuid: "5f6d530faceb42e68529b01bbe675851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6d530faceb42e68529b01bbe675851" member_type: VOTER last_known_addr { host: "127.4.73.2" port: 35951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 } health_report { overall_health: UNKNOWN } } }
I20260810 06:29:21.974108 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:21.970288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.4.73.2:35951)
0810 06:29:21.970421 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:21.970425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:21.970439 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:29:21.970615 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:21.970617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:21.970683 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:21.970790 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:21.971026 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.971629 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.972564 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:21.972827 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:21.972953 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:21.973284 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:21.973702 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:21.973833 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:21.973902 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 06:29:21.973952 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:29:21.979647 6380 raft_consensus.cc:1261] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 1 FOLLOWER]: Refusing update from remote peer 5f6d530faceb42e68529b01bbe675851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:21.979738 5946 raft_consensus.cc:1261] T 804962bf4dd14ae6b70049ae370a9477 P 0dadee1fe26e4ddc899c012bddf0bf3a [term 1 FOLLOWER]: Refusing update from remote peer 5f6d530faceb42e68529b01bbe675851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:21.980389 6510 consensus_queue.cc:1035] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc542c1dbe54ccfbee328c2d723c0d0" member_type: VOTER last_known_addr { host: "127.4.73.3" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:21.980573 6505 consensus_queue.cc:1035] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dadee1fe26e4ddc899c012bddf0bf3a" member_type: VOTER last_known_addr { host: "127.4.73.1" port: 38461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:21.988016 6518 mvcc.cc:205] Tried to move back new op lower bound from 7316862410661416960 to 7316862410564308992. Current Snapshot: MvccSnapshot[applied={T|T < 7316862410661416960}]
I20260810 06:29:21.994982 5723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:21.995136 5723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 06:29:21.997658 5723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8267b78ca584cf894c154b0b2ffe093: Sending DeleteTablet for 3 replicas of tablet 804962bf4dd14ae6b70049ae370a9477
I20260810 06:29:21.998449 6120 tablet_service.cc:1514] Processing DeleteTablet for tablet 804962bf4dd14ae6b70049ae370a9477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:21 UTC) from {username='slave'} at 127.0.0.1:53590
I20260810 06:29:21.998603 5918 tablet_service.cc:1514] Processing DeleteTablet for tablet 804962bf4dd14ae6b70049ae370a9477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:21 UTC) from {username='slave'} at 127.0.0.1:58998
I20260810 06:29:21.998749 6343 tablet_service.cc:1514] Processing DeleteTablet for tablet 804962bf4dd14ae6b70049ae370a9477 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 06:29:21 UTC) from {username='slave'} at 127.0.0.1:43186
I20260810 06:29:21.999330 6536 tablet_replica.cc:330] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: stopping tablet replica
I20260810 06:29:21.999496 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5819
I20260810 06:29:21.999678 6536 raft_consensus.cc:2229] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 06:29:22.002585 6536 raft_consensus.cc:2258] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:22.006283 6536 ts_tablet_manager.cc:1905] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:22.009742 6536 ts_tablet_manager.cc:1918] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:22.009841 6536 log.cc:1196] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-2/wal/wals/804962bf4dd14ae6b70049ae370a9477
I20260810 06:29:22.010133 6536 ts_tablet_manager.cc:1939] T 804962bf4dd14ae6b70049ae370a9477 P 0bc542c1dbe54ccfbee328c2d723c0d0: Deleting consensus metadata
I20260810 06:29:22.011502 5701 catalog_manager.cc:4841] TS 0bc542c1dbe54ccfbee328c2d723c0d0 (127.4.73.3:37847): tablet 804962bf4dd14ae6b70049ae370a9477 (table test-table [id=e4118aa59b2446d6b6a0466a71ae7ca9]) successfully deleted
I20260810 06:29:22.011967 6534 tablet_replica.cc:330] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: stopping tablet replica
I20260810 06:29:22.012354 6534 raft_consensus.cc:2229] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 1 LEADER]: Raft consensus shutting down.
I20260810 06:29:22.012733 6534 raft_consensus.cc:2258] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 06:29:22.015244 6534 ts_tablet_manager.cc:1905] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 06:29:22.018026 6534 ts_tablet_manager.cc:1918] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 06:29:22.018131 6534 log.cc:1196] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Deleting WAL directory at /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786343096561320-4388-0/minicluster-data/ts-1/wal/wals/804962bf4dd14ae6b70049ae370a9477
I20260810 06:29:22.018455 6534 ts_tablet_manager.cc:1939] T 804962bf4dd14ae6b70049ae370a9477 P 5f6d530faceb42e68529b01bbe675851: Deleting consensus metadata
I20260810 06:29:22.019759 5701 catalog_manager.cc:4841] TS 5f6d530faceb42e68529b01bbe675851 (127.4.73.2:35951): tablet 804962bf4dd14ae6b70049ae370a9477 (table test-table [id=e4118aa59b2446d6b6a0466a71ae7ca9]) successfully deleted
W20260810 06:29:22.020427 5701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.73.1:38461 (error 108)
I20260810 06:29:22.020710 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6017
I20260810 06:29:22.027014 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:22.025452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.1:38461 (local address 127.0.0.1:59006)
0810 06:29:22.026863 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:22.026935 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.1:38461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1322}
W20260810 06:29:22.029186 5701 catalog_manager.cc:4584] TS 0dadee1fe26e4ddc899c012bddf0bf3a (127.4.73.1:38461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 804962bf4dd14ae6b70049ae370a9477: Network error: Client connection negotiation failed: client connection to 127.4.73.1:38461: connect: Connection refused (error 111)
I20260810 06:29:22.034894 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6228
I20260810 06:29:22.048296 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 5672
2026-08-10T06:29:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1069 ms)
[----------] 6 tests from EncryptionPolicyTest (11391 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:22 dist-test-slave-45dx krb5kdc[6543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:22 dist-test-slave-45dx krb5kdc[6543](info): set up 2 sockets
Aug 10 06:29:22 dist-test-slave-45dx krb5kdc[6543](info): commencing operation
krb5kdc: starting...
W20260810 06:29:23.304908 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.231s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:23 dist-test-slave-45dx krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:23.427721 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:23.537387 6564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:23.537698 6564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:23.537791 6564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:23.541707 6564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:23.541813 6564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:23.541859 6564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:23.541893 6564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:23.541924 6564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:23.546967 6564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:23.547948 6564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:23.548955 6564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:23.557200 6574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:23.557301 6571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:23.557410 6564 server_base.cc:1034] running on GCE node
W20260810 06:29:23.557296 6570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:23.558089 6564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:23.559190 6564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:23.560384 6564 hybrid_clock.cc:648] HybridClock initialized: now 1786343363560364 us; error 28 us; skew 500 ppm
Aug 10 06:29:23 dist-test-slave-45dx krb5kdc[6543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:23.568295 6564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:23.569346 6564 webserver.cc:460] Webserver started at http://127.0.0.1:42887/ using document root <none> and password file <none>
I20260810 06:29:23.569892 6564 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:23.569953 6564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:23.570127 6564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:23.572216 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "48b3677c0b654e5cae204fbb54173dee"
format_stamp: "Formatted at 2026-08-10 06:29:23 on dist-test-slave-45dx"
server_key: "61f1a53588751fd46ba6498d6e9c2290"
server_key_iv: "c2bb3975a8bc7b7446ab9b3345c4678e"
server_key_version: "encryptionkey@0"
I20260810 06:29:23.572873 6564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "48b3677c0b654e5cae204fbb54173dee"
format_stamp: "Formatted at 2026-08-10 06:29:23 on dist-test-slave-45dx"
server_key: "61f1a53588751fd46ba6498d6e9c2290"
server_key_iv: "c2bb3975a8bc7b7446ab9b3345c4678e"
server_key_version: "encryptionkey@0"
I20260810 06:29:23.576830 6564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:29:23.579392 6591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:23.580664 6564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:23.580766 6564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "48b3677c0b654e5cae204fbb54173dee"
format_stamp: "Formatted at 2026-08-10 06:29:23 on dist-test-slave-45dx"
server_key: "61f1a53588751fd46ba6498d6e9c2290"
server_key_iv: "c2bb3975a8bc7b7446ab9b3345c4678e"
server_key_version: "encryptionkey@0"
I20260810 06:29:23.580850 6564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:23.605559 6564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:23.609225 6564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:23.609390 6564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:23.616853 6564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44855
I20260810 06:29:23.616884 6674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44855 every 8 connection(s)
I20260810 06:29:23.617887 6564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:23.620638 6675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:23.623960 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6564
I20260810 06:29:23.624106 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:23.624472 4388 external_mini_cluster.cc:1428] Setting key 4bdb8f1fa25f35fe418c63a744b608ba
I20260810 06:29:23.626318 6675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee: Bootstrap starting.
I20260810 06:29:23.628546 6675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:23.629256 6675 log.cc:824] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:23.630992 6675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee: No bootstrap required, opened a new log
Aug 10 06:29:23 dist-test-slave-45dx krb5kdc[6543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:29:23.634402 6675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3677c0b654e5cae204fbb54173dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 06:29:23.634652 6675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:23.634719 6675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b3677c0b654e5cae204fbb54173dee, State: Initialized, Role: FOLLOWER
I20260810 06:29:23.635159 6675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3677c0b654e5cae204fbb54173dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 06:29:23.635298 6675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:23.635365 6675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:23.635505 6675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:23.636293 6675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3677c0b654e5cae204fbb54173dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 06:29:23.636682 6675 leader_election.cc:302] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48b3677c0b654e5cae204fbb54173dee; no voters:
I20260810 06:29:23.636956 6675 leader_election.cc:288] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:23.637076 6691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:23.637326 6691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [term 1 LEADER]: Becoming Leader. State: Replica: 48b3677c0b654e5cae204fbb54173dee, State: Running, Role: LEADER
I20260810 06:29:23.637619 6691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3677c0b654e5cae204fbb54173dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } }
I20260810 06:29:23.638217 6675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:23.639578 6692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48b3677c0b654e5cae204fbb54173dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3677c0b654e5cae204fbb54173dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260810 06:29:23.639715 6692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:23.640417 6703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:23.640725 6693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48b3677c0b654e5cae204fbb54173dee. Latest consensus state: current_term: 1 leader_uuid: "48b3677c0b654e5cae204fbb54173dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48b3677c0b654e5cae204fbb54173dee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44855 } } }
I20260810 06:29:23.640870 6693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:23.641474 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:23.625870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56135 (local address 127.0.0.1:44855)
0810 06:29:23.626299 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:23.626315 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:23.626351 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:29:23.627169 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:23.627189 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:23.627514 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:23.627882 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:23.628822 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:23.629935 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:23.630861 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:23.631461 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:23.631649 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:23.634338 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:23.634343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:23.634355 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:23.634385 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:23.636439 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:23.637134 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:23.637142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:23.637147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:23.637284 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:23.637702 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:23.637704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:23.637706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:23.638216 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:23.638442 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:23.640142 (+ 1700us) server_negotiation.cc:299] Negotiation successful
0810 06:29:23.640443 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20260810 06:29:23.643477 6703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:23.644469 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:23.649705 6703 catalog_manager.cc:1347] Generated new cluster ID: 7f7178418afd4a3e9985cfa731d6df5b
I20260810 06:29:23.649775 6703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:23.655797 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:23.646788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55949 (local address 127.0.0.1:44855)
0810 06:29:23.646904 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:23.646908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:23.647159 (+ 251us) server_negotiation.cc:407] Connection header received
0810 06:29:23.647355 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:23.647358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:23.647406 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:23.647492 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:23.648398 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:23.649187 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:23.650198 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:23.650542 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:23.650661 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:23.652102 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:23.652106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:23.652108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:23.652141 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:23.653744 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:23.654347 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:23.654351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:23.654353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:23.654457 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:23.654831 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:23.654835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:23.654837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:23.655024 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:23.655210 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:23.655504 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 06:29:23.655600 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:29:23.667042 6703 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:23.667932 6703 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:23.674757 6703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48b3677c0b654e5cae204fbb54173dee: Generated new TSK 0
I20260810 06:29:23.675426 6703 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:23.682106 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6564
W20260810 06:29:23.695361 6539 connection.cc:535] client connection to 127.0.0.1:44855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44855): Connection reset by peer (error 104)
2026-08-10T06:29:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1635 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:23 dist-test-slave-45dx krb5kdc[6726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:23 dist-test-slave-45dx krb5kdc[6726](info): set up 2 sockets
Aug 10 06:29:23 dist-test-slave-45dx krb5kdc[6726](info): commencing operation
krb5kdc: starting...
W20260810 06:29:24.921048 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:25 dist-test-slave-45dx krb5kdc[6726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:25.046685 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:25.155857 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:25.156116 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:25.156155 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:25.160069 6747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:25.160123 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:25.160142 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:25.160157 6747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:25.160171 6747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:25.165081 6747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:25.165987 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:25.166931 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:25.175431 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:25.175406 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:25.175460 6758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:25.175575 6747 server_base.cc:1034] running on GCE node
I20260810 06:29:25.176344 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:25.177608 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:25.178791 6747 hybrid_clock.cc:648] HybridClock initialized: now 1786343365178769 us; error 37 us; skew 500 ppm
Aug 10 06:29:25 dist-test-slave-45dx krb5kdc[6726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:25.181965 6747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:25.182860 6747 webserver.cc:460] Webserver started at http://127.0.0.1:45439/ using document root <none> and password file <none>
I20260810 06:29:25.183429 6747 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:25.183490 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:25.183718 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:25.185731 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "63d7b53bf9ee46fd9096c492a4c38317"
format_stamp: "Formatted at 2026-08-10 06:29:25 on dist-test-slave-45dx"
server_key: "6e83ba5a879ca52ec72bab6ad25e57ae"
server_key_iv: "1f4467b9f3fad2ba79d6f6eaf1a1d90f"
server_key_version: "encryptionkey@0"
I20260810 06:29:25.186321 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "63d7b53bf9ee46fd9096c492a4c38317"
format_stamp: "Formatted at 2026-08-10 06:29:25 on dist-test-slave-45dx"
server_key: "6e83ba5a879ca52ec72bab6ad25e57ae"
server_key_iv: "1f4467b9f3fad2ba79d6f6eaf1a1d90f"
server_key_version: "encryptionkey@0"
I20260810 06:29:25.189949 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 06:29:25.192190 6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:25.193320 6747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:29:25.193482 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "63d7b53bf9ee46fd9096c492a4c38317"
format_stamp: "Formatted at 2026-08-10 06:29:25 on dist-test-slave-45dx"
server_key: "6e83ba5a879ca52ec72bab6ad25e57ae"
server_key_iv: "1f4467b9f3fad2ba79d6f6eaf1a1d90f"
server_key_version: "encryptionkey@0"
I20260810 06:29:25.193594 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:25.216977 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:25.219792 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:25.220007 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:25.227034 6747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42831
I20260810 06:29:25.227061 6859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42831 every 8 connection(s)
I20260810 06:29:25.227980 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:25.230695 6860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:25.232759 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6747
I20260810 06:29:25.232923 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:25.233204 4388 external_mini_cluster.cc:1428] Setting key 44a99070adb68f04ed018140f8747d84
I20260810 06:29:25.237174 6860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317: Bootstrap starting.
I20260810 06:29:25.239743 6860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:25.240576 6860 log.cc:824] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:25.242784 6860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317: No bootstrap required, opened a new log
I20260810 06:29:25.243252 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:25.234594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57505 (local address 127.0.0.1:42831)
0810 06:29:25.235063 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:25.235081 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:25.235124 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:29:25.236061 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:25.236083 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:25.236510 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:25.236867 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:25.237272 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:25.238777 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:25.239645 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:25.240287 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:25.240482 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:25.241310 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:25.241315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:25.241325 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:25.241849 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:25.241916 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:25.241945 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:25.242087 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:25.242227 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 06:29:25.242350 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":118,"threads_started":1}
I20260810 06:29:25.246179 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:25.246794 6860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7b53bf9ee46fd9096c492a4c38317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42831 } }
I20260810 06:29:25.247005 6860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:25.247072 6860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d7b53bf9ee46fd9096c492a4c38317, State: Initialized, Role: FOLLOWER
I20260810 06:29:25.247520 6860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7b53bf9ee46fd9096c492a4c38317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42831 } }
I20260810 06:29:25.247640 6860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:25.247721 6860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:25.247833 6860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:25.249015 6860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7b53bf9ee46fd9096c492a4c38317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42831 } }
I20260810 06:29:25.249413 6860 leader_election.cc:302] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d7b53bf9ee46fd9096c492a4c38317; no voters:
I20260810 06:29:25.249763 6860 leader_election.cc:288] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:25.249872 6875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:25.250093 6875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [term 1 LEADER]: Becoming Leader. State: Replica: 63d7b53bf9ee46fd9096c492a4c38317, State: Running, Role: LEADER
I20260810 06:29:25.250376 6875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7b53bf9ee46fd9096c492a4c38317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42831 } }
I20260810 06:29:25.250942 6860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:25.252135 6877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63d7b53bf9ee46fd9096c492a4c38317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7b53bf9ee46fd9096c492a4c38317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42831 } } }
I20260810 06:29:25.252117 6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d7b53bf9ee46fd9096c492a4c38317. Latest consensus state: current_term: 1 leader_uuid: "63d7b53bf9ee46fd9096c492a4c38317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7b53bf9ee46fd9096c492a4c38317" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42831 } } }
I20260810 06:29:25.252265 6877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:25.252334 6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:25.252688 6890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:25.253721 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:25.248279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52227 (local address 127.0.0.1:42831)
0810 06:29:25.248431 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:25.248435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:25.248680 (+ 245us) server_negotiation.cc:407] Connection header received
0810 06:29:25.248808 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:25.248810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:25.248860 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:25.248919 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:25.249270 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:25.250051 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:25.251107 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:25.251462 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:25.251569 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:25.252930 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:25.252936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:25.252940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:25.252954 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:25.252984 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:25.253002 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:25.253117 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:25.253522 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 06:29:25.253560 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 06:29:25.256521 6890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:25.262260 6890 catalog_manager.cc:1347] Generated new cluster ID: 4492596bb0a54b11960c47f2993d85be
I20260810 06:29:25.262322 6890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:25.275027 6890 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:25.276014 6890 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:25.282482 6890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63d7b53bf9ee46fd9096c492a4c38317: Generated new TSK 0
I20260810 06:29:25.283087 6890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:25.288527 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6747
W20260810 06:29:25.305158 6724 connection.cc:535] client connection to 127.0.0.1:42831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42831): Connection reset by peer (error 104)
2026-08-10T06:29:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:25 dist-test-slave-45dx krb5kdc[6912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:25 dist-test-slave-45dx krb5kdc[6912](info): set up 2 sockets
Aug 10 06:29:25 dist-test-slave-45dx krb5kdc[6912](info): commencing operation
krb5kdc: starting...
W20260810 06:29:26.549458 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:26 dist-test-slave-45dx krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:26.675725 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:26.792833 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:26.793111 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:26.793190 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:26.796934 6932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:26.797019 6932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:26.797056 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:26.797128 6932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:26.797169 6932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:26.802115 6932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:26.803092 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:26.804057 6932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:26.812285 6942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:26.812326 6943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:26.812340 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:26.812634 6932 server_base.cc:1034] running on GCE node
I20260810 06:29:26.813196 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:26.814185 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:26.815351 6932 hybrid_clock.cc:648] HybridClock initialized: now 1786343366815333 us; error 34 us; skew 500 ppm
Aug 10 06:29:26 dist-test-slave-45dx krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:26.818135 6932 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:26.819028 6932 webserver.cc:460] Webserver started at http://127.0.0.1:38131/ using document root <none> and password file <none>
I20260810 06:29:26.819579 6932 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:26.819639 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:26.819897 6932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:26.822196 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "b92266e76ccf4f59bcc66202a399cf17"
format_stamp: "Formatted at 2026-08-10 06:29:26 on dist-test-slave-45dx"
server_key: "43e19e916c4a56549296bbf793055d54"
server_key_iv: "22406d2ef6c30d87f913b6179337a195"
server_key_version: "encryptionkey@0"
I20260810 06:29:26.822785 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "b92266e76ccf4f59bcc66202a399cf17"
format_stamp: "Formatted at 2026-08-10 06:29:26 on dist-test-slave-45dx"
server_key: "43e19e916c4a56549296bbf793055d54"
server_key_iv: "22406d2ef6c30d87f913b6179337a195"
server_key_version: "encryptionkey@0"
I20260810 06:29:26.826373 6932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:29:26.828729 6962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:26.829941 6932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:26.830087 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "b92266e76ccf4f59bcc66202a399cf17"
format_stamp: "Formatted at 2026-08-10 06:29:26 on dist-test-slave-45dx"
server_key: "43e19e916c4a56549296bbf793055d54"
server_key_iv: "22406d2ef6c30d87f913b6179337a195"
server_key_version: "encryptionkey@0"
I20260810 06:29:26.830201 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:26.834633 6932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:26.837500 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:26.837689 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:26.844826 6932 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43897
I20260810 06:29:26.844842 7058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43897 every 8 connection(s)
I20260810 06:29:26.845642 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:26.848234 7059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:26.852447 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 6932
I20260810 06:29:26.852583 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:26.852893 4388 external_mini_cluster.cc:1428] Setting key 69cbb4bb46607c7eb8bc91ddb92f777e
I20260810 06:29:26.853940 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17: Bootstrap starting.
I20260810 06:29:26.856433 7059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:26.857321 7059 log.cc:824] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:26.858378 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:26.854232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59217 (local address 127.0.0.1:43897)
0810 06:29:26.854714 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:26.854732 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:26.854774 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:29:26.855656 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:26.855678 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:26.855943 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:26.856214 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:26.857014 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:26.857019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:26.857032 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:26.857043 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:26.857096 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:26.857114 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:26.857197 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:26.857363 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 06:29:26.857534 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":146,"threads_started":1}
I20260810 06:29:26.859308 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17: No bootstrap required, opened a new log
I20260810 06:29:26.861114 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:26.863647 7059 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92266e76ccf4f59bcc66202a399cf17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43897 } }
I20260810 06:29:26.863932 7059 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:26.863991 7059 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92266e76ccf4f59bcc66202a399cf17, State: Initialized, Role: FOLLOWER
I20260810 06:29:26.864494 7059 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92266e76ccf4f59bcc66202a399cf17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43897 } }
I20260810 06:29:26.864678 7059 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:26.864748 7059 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:26.864856 7059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:26.865437 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:26.863554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47459 (local address 127.0.0.1:43897)
0810 06:29:26.863732 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:26.863735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:26.863985 (+ 250us) server_negotiation.cc:407] Connection header received
0810 06:29:26.864059 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:26.864064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:26.864105 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:26.864174 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:26.864954 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:26.864959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:26.864962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:26.864976 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:26.865005 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:26.865017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:26.865110 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:26.865267 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 06:29:26.865298 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 06:29:26.865994 7059 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92266e76ccf4f59bcc66202a399cf17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43897 } }
I20260810 06:29:26.866341 7059 leader_election.cc:302] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92266e76ccf4f59bcc66202a399cf17; no voters:
I20260810 06:29:26.866652 7059 leader_election.cc:288] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:26.866715 7076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:26.866981 7076 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [term 1 LEADER]: Becoming Leader. State: Replica: b92266e76ccf4f59bcc66202a399cf17, State: Running, Role: LEADER
I20260810 06:29:26.867306 7076 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92266e76ccf4f59bcc66202a399cf17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43897 } }
I20260810 06:29:26.867830 7059 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:26.868984 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92266e76ccf4f59bcc66202a399cf17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92266e76ccf4f59bcc66202a399cf17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43897 } } }
I20260810 06:29:26.869110 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:26.869509 7086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:26.869475 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92266e76ccf4f59bcc66202a399cf17. Latest consensus state: current_term: 1 leader_uuid: "b92266e76ccf4f59bcc66202a399cf17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92266e76ccf4f59bcc66202a399cf17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43897 } } }
I20260810 06:29:26.870193 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:26.872599 7086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:26.877343 7086 catalog_manager.cc:1347] Generated new cluster ID: 3c5608181e884a5e86d4a8fdbc052879
I20260810 06:29:26.877403 7086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:26.890085 7086 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:26.890362 7086 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:26.899453 7086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92266e76ccf4f59bcc66202a399cf17: Generated new TSK 0
I20260810 06:29:26.900064 7086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:26.906788 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 6932
2026-08-10T06:29:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1608 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:26 dist-test-slave-45dx krb5kdc[7103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:26 dist-test-slave-45dx krb5kdc[7103](info): set up 2 sockets
Aug 10 06:29:26 dist-test-slave-45dx krb5kdc[7103](info): commencing operation
krb5kdc: starting...
W20260810 06:29:28.161116 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:28 dist-test-slave-45dx krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:28.286829 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:28.395664 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:28.395961 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:28.396032 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:28.399955 7126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:28.400041 7126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:28.400079 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:28.400136 7126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:28.400172 7126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:28.405341 7126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:28.406356 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:28.407473 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:28.415951 7137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:28.415977 7139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:28.415951 7136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:28.416282 7126 server_base.cc:1034] running on GCE node
I20260810 06:29:28.416965 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:28.418010 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:28.419204 7126 hybrid_clock.cc:648] HybridClock initialized: now 1786343368419171 us; error 46 us; skew 500 ppm
Aug 10 06:29:28 dist-test-slave-45dx krb5kdc[7103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:28.421857 7126 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:28.422663 7126 webserver.cc:460] Webserver started at http://127.0.0.1:35647/ using document root <none> and password file <none>
I20260810 06:29:28.423149 7126 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:28.423204 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:28.423350 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:28.425326 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "95128efaf19f40dd91d7cff99d761d62"
format_stamp: "Formatted at 2026-08-10 06:29:28 on dist-test-slave-45dx"
server_key: "589164c196c459788e566b55da300c2a"
server_key_iv: "512bdc340e311dad2e98917534d401b8"
server_key_version: "encryptionkey@0"
I20260810 06:29:28.425835 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "95128efaf19f40dd91d7cff99d761d62"
format_stamp: "Formatted at 2026-08-10 06:29:28 on dist-test-slave-45dx"
server_key: "589164c196c459788e566b55da300c2a"
server_key_iv: "512bdc340e311dad2e98917534d401b8"
server_key_version: "encryptionkey@0"
I20260810 06:29:28.429399 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:28.431636 7153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:28.432825 7126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:28.432955 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "95128efaf19f40dd91d7cff99d761d62"
format_stamp: "Formatted at 2026-08-10 06:29:28 on dist-test-slave-45dx"
server_key: "589164c196c459788e566b55da300c2a"
server_key_iv: "512bdc340e311dad2e98917534d401b8"
server_key_version: "encryptionkey@0"
I20260810 06:29:28.433055 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:28.447976 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:28.450718 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:28.450861 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:28.457808 7126 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43867
I20260810 06:29:28.457891 7255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43867 every 8 connection(s)
I20260810 06:29:28.458739 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:28.461544 7257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:28.463100 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7126
I20260810 06:29:28.463260 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:28.463586 4388 external_mini_cluster.cc:1428] Setting key 72bb4eebbcee7352a47c417ff01a2600
I20260810 06:29:28.468578 7257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62: Bootstrap starting.
I20260810 06:29:28.471107 7257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:28.471885 7257 log.cc:824] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:29:28 dist-test-slave-45dx krb5kdc[7103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:29:28.474208 7257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62: No bootstrap required, opened a new log
I20260810 06:29:28.478552 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:28.464828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45267 (local address 127.0.0.1:43867)
0810 06:29:28.465237 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:28.465265 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:28.465308 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:29:28.466107 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:28.466130 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:28.466545 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:28.466907 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:28.467554 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:28.469100 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:28.469926 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:28.470586 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:28.470754 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:28.473284 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:28.473290 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:28.473306 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:28.473341 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:28.475319 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:28.475748 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:28.475755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:28.475760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:28.475911 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:28.476178 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:28.476181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:28.476183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:28.476803 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:28.477046 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:28.477231 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 06:29:28.477533 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":101,"threads_started":1}
I20260810 06:29:28.479074 7257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95128efaf19f40dd91d7cff99d761d62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260810 06:29:28.479362 7257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:28.479487 7257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95128efaf19f40dd91d7cff99d761d62, State: Initialized, Role: FOLLOWER
I20260810 06:29:28.479981 7257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95128efaf19f40dd91d7cff99d761d62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260810 06:29:28.480171 7257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:28.480278 7257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:28.480428 7257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:28.481534 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:28.481563 7257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95128efaf19f40dd91d7cff99d761d62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260810 06:29:28.481950 7257 leader_election.cc:302] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95128efaf19f40dd91d7cff99d761d62; no voters:
I20260810 06:29:28.482267 7257 leader_election.cc:288] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:28.482391 7263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:28.482591 7263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [term 1 LEADER]: Becoming Leader. State: Replica: 95128efaf19f40dd91d7cff99d761d62, State: Running, Role: LEADER
I20260810 06:29:28.482985 7263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95128efaf19f40dd91d7cff99d761d62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } }
I20260810 06:29:28.483538 7257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:28.484177 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95128efaf19f40dd91d7cff99d761d62. Latest consensus state: current_term: 1 leader_uuid: "95128efaf19f40dd91d7cff99d761d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95128efaf19f40dd91d7cff99d761d62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } } }
I20260810 06:29:28.484339 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:28.484671 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95128efaf19f40dd91d7cff99d761d62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95128efaf19f40dd91d7cff99d761d62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43867 } } }
I20260810 06:29:28.484817 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62 [sys.catalog]: This master's current role is: LEADER
W20260810 06:29:28.488997 7296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:28.489079 7296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:28.489146 7281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:28.490031 7281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:28.494596 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:28.483711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58235 (local address 127.0.0.1:43867)
0810 06:29:28.483829 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:28.483833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:28.486257 (+ 2424us) server_negotiation.cc:407] Connection header received
0810 06:29:28.486480 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:28.486483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:28.486526 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:28.486609 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:28.487390 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:28.488088 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:28.489237 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:28.489558 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:28.489653 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:28.491270 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:28.491273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:28.491276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:28.491300 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:28.492957 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:28.493376 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:28.493379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:28.493381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:28.493462 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:28.493744 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:28.493747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:28.493749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:28.493917 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:28.494053 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:28.494288 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 06:29:28.494407 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 06:29:28.496986 7281 catalog_manager.cc:1347] Generated new cluster ID: 8c256000567848f9835ba42c20b49a82
I20260810 06:29:28.497061 7281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:28.513104 7281 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:28.513995 7281 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:28.519956 7281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95128efaf19f40dd91d7cff99d761d62: Generated new TSK 0
I20260810 06:29:28.520618 7281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:28.531901 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7126
2026-08-10T06:29:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1635 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:28 dist-test-slave-45dx krb5kdc[7304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:28 dist-test-slave-45dx krb5kdc[7304](info): set up 2 sockets
Aug 10 06:29:28 dist-test-slave-45dx krb5kdc[7304](info): commencing operation
krb5kdc: starting...
W20260810 06:29:29.789800 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:29 dist-test-slave-45dx krb5kdc[7304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:29.921597 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:30.029565 7326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:30.029860 7326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:30.029939 7326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:30.033879 7326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:30.033973 7326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:30.034015 7326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:30.034062 7326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:30.034096 7326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:30.039203 7326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:30.040230 7326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:30.041239 7326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:30.049656 7338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:30.049690 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:30.049963 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:30.050345 7326 server_base.cc:1034] running on GCE node
I20260810 06:29:30.050863 7326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:30.051878 7326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:30.053037 7326 hybrid_clock.cc:648] HybridClock initialized: now 1786343370053014 us; error 35 us; skew 500 ppm
Aug 10 06:29:30 dist-test-slave-45dx krb5kdc[7304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:30.055660 7326 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:30.056552 7326 webserver.cc:460] Webserver started at http://127.0.0.1:36151/ using document root <none> and password file <none>
I20260810 06:29:30.057045 7326 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:30.057103 7326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:30.057260 7326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:30.059181 7326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "76e6be3a4d784feaab0c85add91c220e"
format_stamp: "Formatted at 2026-08-10 06:29:30 on dist-test-slave-45dx"
server_key: "8270eab449e05fc8ecfc4432508c4db2"
server_key_iv: "d72a05f90e56661e7942ddfd867c4dcc"
server_key_version: "encryptionkey@0"
I20260810 06:29:30.059701 7326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "76e6be3a4d784feaab0c85add91c220e"
format_stamp: "Formatted at 2026-08-10 06:29:30 on dist-test-slave-45dx"
server_key: "8270eab449e05fc8ecfc4432508c4db2"
server_key_iv: "d72a05f90e56661e7942ddfd867c4dcc"
server_key_version: "encryptionkey@0"
I20260810 06:29:30.063359 7326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:29:30.065791 7353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:30.066906 7326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 06:29:30.067040 7326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "76e6be3a4d784feaab0c85add91c220e"
format_stamp: "Formatted at 2026-08-10 06:29:30 on dist-test-slave-45dx"
server_key: "8270eab449e05fc8ecfc4432508c4db2"
server_key_iv: "d72a05f90e56661e7942ddfd867c4dcc"
server_key_version: "encryptionkey@0"
I20260810 06:29:30.067137 7326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:30.083745 7326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:30.086329 7326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:30.086508 7326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:30.093063 7326 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37751
I20260810 06:29:30.093096 7438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37751 every 8 connection(s)
I20260810 06:29:30.093964 7326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:30.096542 7439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:30.097079 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7326
I20260810 06:29:30.097219 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:30.097501 4388 external_mini_cluster.cc:1428] Setting key a85ac09e63ca75e2c6d66e187aa66798
I20260810 06:29:30.102882 7439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e: Bootstrap starting.
I20260810 06:29:30.105104 7439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:30.105834 7439 log.cc:824] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:30.106050 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:30.098888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41425 (local address 127.0.0.1:37751)
0810 06:29:30.099304 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:30.099319 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:30.099352 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:29:30.099999 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:30.100015 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:30.100332 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:30.100635 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:30.101026 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:30.101989 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:30.102851 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:30.103352 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:30.103519 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:30.104386 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:30.104390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:30.104407 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:30.104753 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:30.104804 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:30.104826 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:30.104933 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:30.105100 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 06:29:30.105306 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260810 06:29:30.107820 7439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e: No bootstrap required, opened a new log
I20260810 06:29:30.108701 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:30.112407 7439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e6be3a4d784feaab0c85add91c220e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260810 06:29:30.112650 7439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:30.112718 7439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e6be3a4d784feaab0c85add91c220e, State: Initialized, Role: FOLLOWER
I20260810 06:29:30.113129 7439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e6be3a4d784feaab0c85add91c220e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260810 06:29:30.113287 7439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:30.113358 7439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:30.113445 7439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:30.114596 7439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e6be3a4d784feaab0c85add91c220e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260810 06:29:30.114791 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:30.110806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59867 (local address 127.0.0.1:37751)
0810 06:29:30.110916 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:30.110922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:30.111124 (+ 202us) server_negotiation.cc:407] Connection header received
0810 06:29:30.111196 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:30.111199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:30.111240 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:30.111322 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:30.111543 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:30.112269 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:30.113223 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:30.113591 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:30.113704 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:30.114403 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:30.114408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:30.114410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:30.114425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:30.114457 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:30.114474 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:30.114557 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:30.114616 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 06:29:30.114665 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:29:30.114985 7439 leader_election.cc:302] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e6be3a4d784feaab0c85add91c220e; no voters:
I20260810 06:29:30.115288 7439 leader_election.cc:288] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:30.115399 7457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:30.115604 7457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [term 1 LEADER]: Becoming Leader. State: Replica: 76e6be3a4d784feaab0c85add91c220e, State: Running, Role: LEADER
I20260810 06:29:30.115985 7457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e6be3a4d784feaab0c85add91c220e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } }
I20260810 06:29:30.116514 7439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:30.117515 7459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76e6be3a4d784feaab0c85add91c220e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e6be3a4d784feaab0c85add91c220e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } } }
I20260810 06:29:30.117727 7459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:30.118026 7459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76e6be3a4d784feaab0c85add91c220e. Latest consensus state: current_term: 1 leader_uuid: "76e6be3a4d784feaab0c85add91c220e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e6be3a4d784feaab0c85add91c220e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37751 } } }
I20260810 06:29:30.118150 7476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:30.118180 7459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:30.120790 7476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:30.126151 7476 catalog_manager.cc:1347] Generated new cluster ID: 86157e3164c84e6880763d18358dea44
I20260810 06:29:30.126215 7476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:30.143687 7476 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:30.144761 7476 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:30.151124 7476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76e6be3a4d784feaab0c85add91c220e: Generated new TSK 0
I20260810 06:29:30.151716 7476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:30.155921 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7326
2026-08-10T06:29:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:30 dist-test-slave-45dx krb5kdc[7497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:30 dist-test-slave-45dx krb5kdc[7497](info): set up 2 sockets
Aug 10 06:29:30 dist-test-slave-45dx krb5kdc[7497](info): commencing operation
krb5kdc: starting...
W20260810 06:29:31.434719 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:31 dist-test-slave-45dx krb5kdc[7497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:31.558383 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:31.667444 7519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:31.667717 7519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:31.667790 7519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:31.671756 7519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:31.671835 7519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:31.671873 7519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:31.671926 7519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:31.671967 7519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:31.677090 7519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:31.678112 7519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:31.679103 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:31.687737 7530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:31.687737 7533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:31.687778 7519 server_base.cc:1034] running on GCE node
W20260810 06:29:31.687731 7531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:31.688633 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:31.689741 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:31.690907 7519 hybrid_clock.cc:648] HybridClock initialized: now 1786343371690886 us; error 31 us; skew 500 ppm
Aug 10 06:29:31 dist-test-slave-45dx krb5kdc[7497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:31.693853 7519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:31.694729 7519 webserver.cc:460] Webserver started at http://127.0.0.1:41213/ using document root <none> and password file <none>
I20260810 06:29:31.695313 7519 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:31.695406 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:31.695643 7519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:31.697750 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "ae7417169ea74017af949131d8886fa7"
format_stamp: "Formatted at 2026-08-10 06:29:31 on dist-test-slave-45dx"
server_key: "f748ff4ca75df26d2cc1137a66633c07"
server_key_iv: "88cdc300ae5cc19f77a5950aeba9ef8d"
server_key_version: "encryptionkey@0"
I20260810 06:29:31.698340 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "ae7417169ea74017af949131d8886fa7"
format_stamp: "Formatted at 2026-08-10 06:29:31 on dist-test-slave-45dx"
server_key: "f748ff4ca75df26d2cc1137a66633c07"
server_key_iv: "88cdc300ae5cc19f77a5950aeba9ef8d"
server_key_version: "encryptionkey@0"
I20260810 06:29:31.702014 7519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:31.704466 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:31.705638 7519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:31.705785 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "ae7417169ea74017af949131d8886fa7"
format_stamp: "Formatted at 2026-08-10 06:29:31 on dist-test-slave-45dx"
server_key: "f748ff4ca75df26d2cc1137a66633c07"
server_key_iv: "88cdc300ae5cc19f77a5950aeba9ef8d"
server_key_version: "encryptionkey@0"
I20260810 06:29:31.705894 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:31.710434 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:31.713196 7519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:31.713378 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:31.720431 7519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45849
I20260810 06:29:31.720489 7637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45849 every 8 connection(s)
I20260810 06:29:31.721334 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:31.724000 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7519
I20260810 06:29:31.724032 7638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:31.724161 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:31.724502 4388 external_mini_cluster.cc:1428] Setting key dd62d5668d77d84706eb39504c49162d
I20260810 06:29:31.729645 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:31.725846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49939 (local address 127.0.0.1:45849)
0810 06:29:31.726287 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:31.726300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:31.726335 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:29:31.727156 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:31.727174 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:31.727446 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:31.727766 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:31.728564 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:31.728568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:31.728580 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:31.728590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:31.728642 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:31.728661 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:31.728805 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:31.728879 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 06:29:31.729022 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260810 06:29:31.730432 7638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7: Bootstrap starting.
I20260810 06:29:31.732414 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:31.732923 7638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:31.733675 7638 log.cc:824] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:31.735961 7638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7: No bootstrap required, opened a new log
I20260810 06:29:31.736089 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:31.734420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42847 (local address 127.0.0.1:45849)
0810 06:29:31.734535 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:31.734539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:31.734738 (+ 199us) server_negotiation.cc:407] Connection header received
0810 06:29:31.734852 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:31.734854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:31.734881 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:31.734945 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:31.735608 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:31.735610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:31.735612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:31.735623 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:31.735646 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:31.735656 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:31.735723 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:31.735848 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 06:29:31.735879 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:29:31.739750 7638 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7417169ea74017af949131d8886fa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 06:29:31.739993 7638 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:31.740052 7638 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7417169ea74017af949131d8886fa7, State: Initialized, Role: FOLLOWER
I20260810 06:29:31.740486 7638 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7417169ea74017af949131d8886fa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 06:29:31.740638 7638 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:31.740705 7638 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:31.740801 7638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:31.741717 7638 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7417169ea74017af949131d8886fa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 06:29:31.742069 7638 leader_election.cc:302] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae7417169ea74017af949131d8886fa7; no voters:
I20260810 06:29:31.742439 7638 leader_election.cc:288] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:31.742532 7658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:31.742810 7658 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [term 1 LEADER]: Becoming Leader. State: Replica: ae7417169ea74017af949131d8886fa7, State: Running, Role: LEADER
I20260810 06:29:31.743176 7658 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7417169ea74017af949131d8886fa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } }
I20260810 06:29:31.743752 7638 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:31.745018 7660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae7417169ea74017af949131d8886fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7417169ea74017af949131d8886fa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260810 06:29:31.745035 7659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae7417169ea74017af949131d8886fa7. Latest consensus state: current_term: 1 leader_uuid: "ae7417169ea74017af949131d8886fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7417169ea74017af949131d8886fa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45849 } } }
I20260810 06:29:31.745240 7659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:31.745458 7660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:31.746073 7671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:31.749322 7671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:31.754240 7671 catalog_manager.cc:1347] Generated new cluster ID: 8adc6acbb33847ebaf015e65b7e0f0c1
I20260810 06:29:31.754309 7671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:31.764274 7671 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:31.764595 7671 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:31.769131 7671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae7417169ea74017af949131d8886fa7: Generated new TSK 0
I20260810 06:29:31.769711 7671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:31.777635 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7519
2026-08-10T06:29:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:31 dist-test-slave-45dx krb5kdc[7689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:31 dist-test-slave-45dx krb5kdc[7689](info): set up 2 sockets
Aug 10 06:29:31 dist-test-slave-45dx krb5kdc[7689](info): commencing operation
krb5kdc: starting...
W20260810 06:29:33.044445 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:33 dist-test-slave-45dx krb5kdc[7689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:33.165104 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:33.274189 7713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:33.274415 7713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:33.274449 7713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:33.278302 7713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:33.278353 7713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:33.278376 7713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:33.278424 7713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:33.278453 7713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:33.283459 7713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:33.284473 7713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:33.285394 7713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:33.293262 7720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:33.293282 7723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:33.293262 7719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:33.293447 7713 server_base.cc:1034] running on GCE node
I20260810 06:29:33.294076 7713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:33.295012 7713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:33.296197 7713 hybrid_clock.cc:648] HybridClock initialized: now 1786343373296185 us; error 32 us; skew 500 ppm
Aug 10 06:29:33 dist-test-slave-45dx krb5kdc[7689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:33.298782 7713 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:33.299577 7713 webserver.cc:460] Webserver started at http://127.0.0.1:44455/ using document root <none> and password file <none>
I20260810 06:29:33.300096 7713 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:33.300180 7713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:33.300450 7713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:33.302349 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "df4fcbb8403a415e8f0e89031352a49e"
format_stamp: "Formatted at 2026-08-10 06:29:33 on dist-test-slave-45dx"
server_key: "f6d352c65658d91a597c6a74e214b403"
server_key_iv: "ade37f264d55f3c363eb6e0eaf1bd315"
server_key_version: "encryptionkey@0"
I20260810 06:29:33.302901 7713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "df4fcbb8403a415e8f0e89031352a49e"
format_stamp: "Formatted at 2026-08-10 06:29:33 on dist-test-slave-45dx"
server_key: "f6d352c65658d91a597c6a74e214b403"
server_key_iv: "ade37f264d55f3c363eb6e0eaf1bd315"
server_key_version: "encryptionkey@0"
I20260810 06:29:33.306367 7713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:33.308506 7738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:33.309629 7713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:33.309782 7713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "df4fcbb8403a415e8f0e89031352a49e"
format_stamp: "Formatted at 2026-08-10 06:29:33 on dist-test-slave-45dx"
server_key: "f6d352c65658d91a597c6a74e214b403"
server_key_iv: "ade37f264d55f3c363eb6e0eaf1bd315"
server_key_version: "encryptionkey@0"
I20260810 06:29:33.309887 7713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:33.326041 7713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:33.328908 7713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:33.329113 7713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:33.336004 7713 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37775
I20260810 06:29:33.336031 7827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37775 every 8 connection(s)
I20260810 06:29:33.336949 7713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:33.339555 7829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:33.340962 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7713
I20260810 06:29:33.341100 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:33.341444 4388 external_mini_cluster.cc:1428] Setting key dcf978ec7c72f3307356405ec83e9e29
I20260810 06:29:33.345590 7829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e: Bootstrap starting.
I20260810 06:29:33.347563 7829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:33.348271 7829 log.cc:824] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:29:33 dist-test-slave-45dx krb5kdc[7689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:29:33.350461 7829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e: No bootstrap required, opened a new log
I20260810 06:29:33.354610 7829 raft_consensus.cc:348] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4fcbb8403a415e8f0e89031352a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260810 06:29:33.354907 7829 raft_consensus.cc:374] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:33.354981 7829 raft_consensus.cc:729] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4fcbb8403a415e8f0e89031352a49e, State: Initialized, Role: FOLLOWER
I20260810 06:29:33.355377 7829 consensus_queue.cc:260] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4fcbb8403a415e8f0e89031352a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260810 06:29:33.355515 7829 raft_consensus.cc:388] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:33.355592 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:33.355681 7829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:33.355693 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:33.342971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:57551 (local address 127.0.0.1:37775)
0810 06:29:33.343399 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:33.343413 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:33.343448 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:29:33.344052 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:33.344067 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:33.344447 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:33.344751 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:33.345549 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:33.346485 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:33.347278 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:33.347762 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:33.347925 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:33.350887 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:33.350892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:33.350904 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:33.350929 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:33.352662 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:33.353225 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:33.353229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:33.353233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:33.353357 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:33.353705 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:33.353707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:33.353708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:33.354086 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:33.354255 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:33.354559 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 06:29:33.354824 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":131,"threads_started":1}
I20260810 06:29:33.356695 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4fcbb8403a415e8f0e89031352a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260810 06:29:33.356992 7829 leader_election.cc:302] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df4fcbb8403a415e8f0e89031352a49e; no voters:
I20260810 06:29:33.357275 7829 leader_election.cc:288] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:33.357368 7837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:33.357574 7837 raft_consensus.cc:686] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [term 1 LEADER]: Becoming Leader. State: Replica: df4fcbb8403a415e8f0e89031352a49e, State: Running, Role: LEADER
I20260810 06:29:33.357961 7837 consensus_queue.cc:237] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4fcbb8403a415e8f0e89031352a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } }
I20260810 06:29:33.358582 7829 sys_catalog.cc:564] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:33.359073 7837 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader df4fcbb8403a415e8f0e89031352a49e. Latest consensus state: current_term: 1 leader_uuid: "df4fcbb8403a415e8f0e89031352a49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4fcbb8403a415e8f0e89031352a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } } }
I20260810 06:29:33.359184 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:33.359850 7837 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:33.359730 7838 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df4fcbb8403a415e8f0e89031352a49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4fcbb8403a415e8f0e89031352a49e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37775 } } }
I20260810 06:29:33.360047 7838 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:33.360800 7853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:33.363941 7853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:33.368970 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:33.361534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:54131 (local address 127.0.0.1:37775)
0810 06:29:33.361651 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:33.361656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:33.361822 (+ 166us) server_negotiation.cc:407] Connection header received
0810 06:29:33.361905 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:33.361907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:33.361951 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:33.362035 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:33.362621 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:33.363117 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:33.363855 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:33.364120 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:33.364222 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:33.365487 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:33.365491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:33.365494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:33.365516 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:33.367033 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:33.367564 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:33.367567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:33.367568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:33.367652 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:33.367974 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:33.367977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:33.367979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:33.368159 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:33.368309 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:33.368730 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 06:29:33.368829 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 06:29:33.369691 7853 catalog_manager.cc:1347] Generated new cluster ID: 1038c2e3dd554c658d1eedd2906ad349
I20260810 06:29:33.369836 7853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:33.387336 7853 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:33.388187 7853 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:33.395782 7853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df4fcbb8403a415e8f0e89031352a49e: Generated new TSK 0
I20260810 06:29:33.396488 7853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:33.402321 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7713
W20260810 06:29:33.419869 7686 connection.cc:535] client connection to 127.0.0.1:37775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37775): Connection reset by peer (error 104)
2026-08-10T06:29:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:33 dist-test-slave-45dx krb5kdc[7879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:33 dist-test-slave-45dx krb5kdc[7879](info): set up 2 sockets
Aug 10 06:29:33 dist-test-slave-45dx krb5kdc[7879](info): commencing operation
krb5kdc: starting...
W20260810 06:29:34.666033 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.232s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:34 dist-test-slave-45dx krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:34.787460 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:34.896751 7897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:34.896983 7897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:34.897019 7897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:34.900882 7897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:34.900933 7897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:34.900951 7897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:34.901000 7897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:34.901031 7897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:34.905967 7897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:34.906919 7897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:34.907929 7897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:34.916160 7909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:34.916213 7912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:34.916356 7897 server_base.cc:1034] running on GCE node
W20260810 06:29:34.916173 7906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:34.917065 7897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:34.918077 7897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:34.919248 7897 hybrid_clock.cc:648] HybridClock initialized: now 1786343374919225 us; error 31 us; skew 500 ppm
Aug 10 06:29:34 dist-test-slave-45dx krb5kdc[7879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:34.928262 7897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:34.929244 7897 webserver.cc:460] Webserver started at http://127.0.0.1:45921/ using document root <none> and password file <none>
I20260810 06:29:34.929760 7897 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:34.929819 7897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:34.930019 7897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:34.931948 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "c0aef803f58644b1af016fddb2cf5ca0"
format_stamp: "Formatted at 2026-08-10 06:29:34 on dist-test-slave-45dx"
server_key: "188984c41bfd8c57c63a51311ea9e5c2"
server_key_iv: "170249963b9c6ff8112e3cf05340a93d"
server_key_version: "encryptionkey@0"
I20260810 06:29:34.932539 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "c0aef803f58644b1af016fddb2cf5ca0"
format_stamp: "Formatted at 2026-08-10 06:29:34 on dist-test-slave-45dx"
server_key: "188984c41bfd8c57c63a51311ea9e5c2"
server_key_iv: "170249963b9c6ff8112e3cf05340a93d"
server_key_version: "encryptionkey@0"
I20260810 06:29:34.936127 7897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:29:34.938519 7922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:34.939688 7897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:34.939841 7897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "c0aef803f58644b1af016fddb2cf5ca0"
format_stamp: "Formatted at 2026-08-10 06:29:34 on dist-test-slave-45dx"
server_key: "188984c41bfd8c57c63a51311ea9e5c2"
server_key_iv: "170249963b9c6ff8112e3cf05340a93d"
server_key_version: "encryptionkey@0"
I20260810 06:29:34.939949 7897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:34.963798 7897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:34.967003 7897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:34.967196 7897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:34.974200 7897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45557
I20260810 06:29:34.974212 8021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45557 every 8 connection(s)
I20260810 06:29:34.975080 7897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:34.977716 8022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:34.982858 8022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0: Bootstrap starting.
I20260810 06:29:34.983726 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 7897
I20260810 06:29:34.983873 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:34.984222 4388 external_mini_cluster.cc:1428] Setting key 32a3aeee31d7a67dec107b1b3483cfe8
I20260810 06:29:34.985654 8022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:34.986337 8022 log.cc:824] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:34.988296 8022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0: No bootstrap required, opened a new log
I20260810 06:29:34.992501 8022 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aef803f58644b1af016fddb2cf5ca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260810 06:29:34.992694 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:34.985297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:43823 (local address 127.0.0.1:45557)
0810 06:29:34.985688 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:34.985729 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:34.985775 (+ 46us) server_negotiation.cc:407] Connection header received
0810 06:29:34.986345 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:34.986360 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:34.986743 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:34.987094 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:34.987395 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:34.988447 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:34.989655 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:34.990128 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:34.990236 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:34.991037 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:34.991042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:34.991052 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:34.991395 (+ 343us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:34.991444 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:34.991464 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:34.991592 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:34.991790 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 06:29:34.991953 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":116,"threads_started":1}
I20260810 06:29:34.992699 8022 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:34.992830 8022 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0aef803f58644b1af016fddb2cf5ca0, State: Initialized, Role: FOLLOWER
I20260810 06:29:34.993260 8022 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aef803f58644b1af016fddb2cf5ca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260810 06:29:34.993383 8022 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:34.993448 8022 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:34.993557 8022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:34.994524 8022 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aef803f58644b1af016fddb2cf5ca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260810 06:29:34.994874 8022 leader_election.cc:302] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0aef803f58644b1af016fddb2cf5ca0; no voters:
I20260810 06:29:34.995163 8022 leader_election.cc:288] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:34.995237 8029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:34.995471 8029 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [term 1 LEADER]: Becoming Leader. State: Replica: c0aef803f58644b1af016fddb2cf5ca0, State: Running, Role: LEADER
I20260810 06:29:34.995564 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:34.995775 8029 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aef803f58644b1af016fddb2cf5ca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } }
I20260810 06:29:34.996351 8022 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:34.997171 8031 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0aef803f58644b1af016fddb2cf5ca0. Latest consensus state: current_term: 1 leader_uuid: "c0aef803f58644b1af016fddb2cf5ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aef803f58644b1af016fddb2cf5ca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } } }
I20260810 06:29:34.997284 8031 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:34.997581 8030 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0aef803f58644b1af016fddb2cf5ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0aef803f58644b1af016fddb2cf5ca0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45557 } } }
I20260810 06:29:34.997656 8030 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:34.997648 8045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:35.001329 8045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:35.003031 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:34.998054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:41073 (local address 127.0.0.1:45557)
0810 06:29:34.998147 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:34.998151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:34.998861 (+ 710us) server_negotiation.cc:407] Connection header received
0810 06:29:34.999007 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:34.999010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:34.999056 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:34.999127 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:34.999399 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:35.000162 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:35.000973 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:35.001310 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:35.001401 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:35.002557 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:35.002561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:35.002563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:35.002575 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:35.002605 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:35.002616 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:35.002706 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:35.002830 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 06:29:35.002860 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 06:29:35.006712 8045 catalog_manager.cc:1347] Generated new cluster ID: 94c807c174894181ba0c7b75a06a9db6
I20260810 06:29:35.006774 8045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:35.016469 8045 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:35.017246 8045 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:35.031788 8045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0aef803f58644b1af016fddb2cf5ca0: Generated new TSK 0
I20260810 06:29:35.032483 8045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:35.041168 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 7897
W20260810 06:29:35.052139 7876 connection.cc:535] client connection to 127.0.0.1:45557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45557): Connection reset by peer (error 104)
2026-08-10T06:29:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1632 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:35 dist-test-slave-45dx krb5kdc[8066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:35 dist-test-slave-45dx krb5kdc[8066](info): set up 2 sockets
Aug 10 06:29:35 dist-test-slave-45dx krb5kdc[8066](info): commencing operation
krb5kdc: starting...
W20260810 06:29:36.284565 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:36 dist-test-slave-45dx krb5kdc[8066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:36.407657 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36923
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:36.517071 8087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:36.517321 8087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:36.517383 8087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:36.521143 8087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:36.521214 8087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:36.521250 8087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:36.521310 8087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:36.521347 8087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:36.525995 8087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:36.526855 8087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:36.527731 8087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:36.535737 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:36.535768 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:36.535862 8087 server_base.cc:1034] running on GCE node
W20260810 06:29:36.535744 8098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:36.536579 8087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:36.537587 8087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:36.538738 8087 hybrid_clock.cc:648] HybridClock initialized: now 1786343376538721 us; error 28 us; skew 500 ppm
Aug 10 06:29:36 dist-test-slave-45dx krb5kdc[8066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:36.541420 8087 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:36.542173 8087 webserver.cc:460] Webserver started at http://127.0.0.1:35987/ using document root <none> and password file <none>
I20260810 06:29:36.542649 8087 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:36.542704 8087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:36.542865 8087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:36.544742 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "594ceaf29b75438ab71a6c3662c9dcbb"
format_stamp: "Formatted at 2026-08-10 06:29:36 on dist-test-slave-45dx"
server_key: "c255f7e30ea39bfab03a71da57fa4eb6"
server_key_iv: "cbe090907368bbefa55afc7953b86b4b"
server_key_version: "encryptionkey@0"
I20260810 06:29:36.545230 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "594ceaf29b75438ab71a6c3662c9dcbb"
format_stamp: "Formatted at 2026-08-10 06:29:36 on dist-test-slave-45dx"
server_key: "c255f7e30ea39bfab03a71da57fa4eb6"
server_key_iv: "cbe090907368bbefa55afc7953b86b4b"
server_key_version: "encryptionkey@0"
I20260810 06:29:36.548501 8087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 06:29:36.550706 8109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:36.551880 8087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:36.552021 8087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "594ceaf29b75438ab71a6c3662c9dcbb"
format_stamp: "Formatted at 2026-08-10 06:29:36 on dist-test-slave-45dx"
server_key: "c255f7e30ea39bfab03a71da57fa4eb6"
server_key_iv: "cbe090907368bbefa55afc7953b86b4b"
server_key_version: "encryptionkey@0"
I20260810 06:29:36.552131 8087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:36.556134 8087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:36.558764 8087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:36.558977 8087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:36.565478 8087 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36923
I20260810 06:29:36.565510 8197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36923 every 8 connection(s)
I20260810 06:29:36.566265 8087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:36.568707 8200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:36.573597 8200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb: Bootstrap starting.
I20260810 06:29:36.574002 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8087
I20260810 06:29:36.574132 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:36.574424 4388 external_mini_cluster.cc:1428] Setting key e87fddc92489b1d09a105bf07dd0649c
I20260810 06:29:36.576115 8200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:36.576850 8200 log.cc:824] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:36.578824 8200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb: No bootstrap required, opened a new log
I20260810 06:29:36.579082 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:36.575565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:33701 (local address 127.0.0.1:36923)
0810 06:29:36.575974 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:36.575987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:36.576018 (+ 31us) server_negotiation.cc:407] Connection header received
0810 06:29:36.576685 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:36.576702 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:36.576993 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:36.577293 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:36.577949 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:36.577954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:36.577973 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:36.577982 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:36.578030 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:36.578048 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:36.578157 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:36.578254 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 06:29:36.578404 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260810 06:29:36.581851 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:36.582870 8200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260810 06:29:36.583128 8200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:36.583184 8200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594ceaf29b75438ab71a6c3662c9dcbb, State: Initialized, Role: FOLLOWER
I20260810 06:29:36.583575 8200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260810 06:29:36.583738 8200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:36.583797 8200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:36.583859 8200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:36.584734 8200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260810 06:29:36.585098 8200 leader_election.cc:302] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 594ceaf29b75438ab71a6c3662c9dcbb; no voters:
I20260810 06:29:36.585438 8200 leader_election.cc:288] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:36.585533 8216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:36.585768 8216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [term 1 LEADER]: Becoming Leader. State: Replica: 594ceaf29b75438ab71a6c3662c9dcbb, State: Running, Role: LEADER
I20260810 06:29:36.586004 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:36.583678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:50157 (local address 127.0.0.1:36923)
0810 06:29:36.583788 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:36.583793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:36.584571 (+ 778us) server_negotiation.cc:407] Connection header received
0810 06:29:36.584694 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:36.584696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:36.584720 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:36.584792 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:36.585519 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:36.585522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:36.585525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:36.585539 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:36.585565 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:36.585579 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:36.585665 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:36.585869 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 06:29:36.585908 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:29:36.586426 8216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } }
I20260810 06:29:36.586634 8200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:36.588114 8217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } } }
I20260810 06:29:36.588234 8217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:36.588114 8218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 594ceaf29b75438ab71a6c3662c9dcbb. Latest consensus state: current_term: 1 leader_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "594ceaf29b75438ab71a6c3662c9dcbb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36923 } } }
I20260810 06:29:36.588560 8218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb [sys.catalog]: This master's current role is: LEADER
W20260810 06:29:36.590857 8236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:36.590925 8236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:36.591178 8234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:36.592072 8234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:36.596869 8234 catalog_manager.cc:1347] Generated new cluster ID: b4fd699d0d9542449368f9505a518820
I20260810 06:29:36.596927 8234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:36.608043 8234 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:36.608363 8234 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:36.613955 8234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 594ceaf29b75438ab71a6c3662c9dcbb: Generated new TSK 0
I20260810 06:29:36.614675 8234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:36.622123 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8087
2026-08-10T06:29:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1585 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:36 dist-test-slave-45dx krb5kdc[8245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:36 dist-test-slave-45dx krb5kdc[8245](info): set up 2 sockets
Aug 10 06:29:36 dist-test-slave-45dx krb5kdc[8245](info): commencing operation
krb5kdc: starting...
W20260810 06:29:37.880687 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:37 dist-test-slave-45dx krb5kdc[8245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:38.005303 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:38.115110 8269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:38.115397 8269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:38.115463 8269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:38.119346 8269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:38.119429 8269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:38.119468 8269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:38.119520 8269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:38.119560 8269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:38.124696 8269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:38.125689 8269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:38.126693 8269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:38.134961 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:38.135038 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:38.135129 8269 server_base.cc:1034] running on GCE node
W20260810 06:29:38.134977 8275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:38.135809 8269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:38.136849 8269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:38.138013 8269 hybrid_clock.cc:648] HybridClock initialized: now 1786343378137986 us; error 38 us; skew 500 ppm
Aug 10 06:29:38 dist-test-slave-45dx krb5kdc[8245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:38.140763 8269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:38.141547 8269 webserver.cc:460] Webserver started at http://127.0.0.1:44973/ using document root <none> and password file <none>
I20260810 06:29:38.142051 8269 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:38.142108 8269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:38.142271 8269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:38.144145 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "c58b966d4040438a97b516da578b5e27"
format_stamp: "Formatted at 2026-08-10 06:29:38 on dist-test-slave-45dx"
server_key: "ccbf798913dd0b76c2c62f0cdc4de7dc"
server_key_iv: "c1b1a11f1cda213e5c6a45778f1bdbaf"
server_key_version: "encryptionkey@0"
I20260810 06:29:38.144703 8269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "c58b966d4040438a97b516da578b5e27"
format_stamp: "Formatted at 2026-08-10 06:29:38 on dist-test-slave-45dx"
server_key: "ccbf798913dd0b76c2c62f0cdc4de7dc"
server_key_iv: "c1b1a11f1cda213e5c6a45778f1bdbaf"
server_key_version: "encryptionkey@0"
I20260810 06:29:38.148128 8269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:29:38.150480 8296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:38.151664 8269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:38.151815 8269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "c58b966d4040438a97b516da578b5e27"
format_stamp: "Formatted at 2026-08-10 06:29:38 on dist-test-slave-45dx"
server_key: "ccbf798913dd0b76c2c62f0cdc4de7dc"
server_key_iv: "c1b1a11f1cda213e5c6a45778f1bdbaf"
server_key_version: "encryptionkey@0"
I20260810 06:29:38.151921 8269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:38.162451 8269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:38.165158 8269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:38.165335 8269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:38.171934 8269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34983
I20260810 06:29:38.171952 8380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34983 every 8 connection(s)
I20260810 06:29:38.172853 8269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:38.175387 8385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:38.180542 8385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27: Bootstrap starting.
I20260810 06:29:38.181429 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8269
I20260810 06:29:38.181591 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:38.181943 4388 external_mini_cluster.cc:1428] Setting key e69553a339f7215ce8ec0526f667cdf6
I20260810 06:29:38.183403 8385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:38.184108 8385 log.cc:824] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:38.185861 8385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27: No bootstrap required, opened a new log
I20260810 06:29:38.189150 8385 raft_consensus.cc:348] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58b966d4040438a97b516da578b5e27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34983 } }
I20260810 06:29:38.189452 8385 raft_consensus.cc:374] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:38.189533 8385 raft_consensus.cc:729] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58b966d4040438a97b516da578b5e27, State: Initialized, Role: FOLLOWER
I20260810 06:29:38.189989 8385 consensus_queue.cc:260] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58b966d4040438a97b516da578b5e27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34983 } }
I20260810 06:29:38.190136 8385 raft_consensus.cc:388] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:38.190227 8385 raft_consensus.cc:482] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:38.190320 8385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 06:29:38 dist-test-slave-45dx krb5kdc[8245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 06:29:38.191442 8385 raft_consensus.cc:504] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58b966d4040438a97b516da578b5e27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34983 } }
I20260810 06:29:38.191792 8385 leader_election.cc:302] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c58b966d4040438a97b516da578b5e27; no voters:
I20260810 06:29:38.192103 8385 leader_election.cc:288] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:38.192224 8392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:38.192523 8392 raft_consensus.cc:686] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [term 1 LEADER]: Becoming Leader. State: Replica: c58b966d4040438a97b516da578b5e27, State: Running, Role: LEADER
I20260810 06:29:38.192893 8392 consensus_queue.cc:237] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58b966d4040438a97b516da578b5e27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34983 } }
I20260810 06:29:38.193379 8385 sys_catalog.cc:564] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:38.194592 8396 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c58b966d4040438a97b516da578b5e27. Latest consensus state: current_term: 1 leader_uuid: "c58b966d4040438a97b516da578b5e27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58b966d4040438a97b516da578b5e27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34983 } } }
I20260810 06:29:38.194698 8395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c58b966d4040438a97b516da578b5e27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58b966d4040438a97b516da578b5e27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34983 } } }
I20260810 06:29:38.194803 8395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:38.194803 8396 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27 [sys.catalog]: This master's current role is: LEADER
W20260810 06:29:38.198050 8415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 06:29:38.198109 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:38.183048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:49537 (local address 127.0.0.1:34983)
0810 06:29:38.183566 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:38.183582 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:38.183625 (+ 43us) server_negotiation.cc:407] Connection header received
0810 06:29:38.184399 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:38.184419 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:38.184746 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:38.185089 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:38.185982 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:38.187010 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:38.187937 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:38.188518 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:38.188681 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:38.191966 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:38.191972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:38.191985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:38.192013 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:38.194302 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:38.194927 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:38.194938 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:38.194947 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:38.195156 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:38.195593 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:38.195596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:38.195598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:38.196100 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:38.196339 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:38.196864 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 06:29:38.197117 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":117,"threads_started":1}
W20260810 06:29:38.198138 8415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:38.198823 8414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:38.199654 8414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:38.202610 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:38.205405 8414 catalog_manager.cc:1347] Generated new cluster ID: c81b029f4ab14982acfcf7c2c841ccf1
I20260810 06:29:38.205463 8414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:38.214521 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:38.205138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:48827 (local address 127.0.0.1:34983)
0810 06:29:38.205255 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:38.205260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:38.205275 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:29:38.205315 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:38.205320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:38.205370 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:38.205455 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:38.206445 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:38.207220 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:38.208527 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:38.208877 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:38.208968 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:38.210819 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:38.210823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:38.210826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:38.210850 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:38.212556 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:38.213159 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:38.213162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:38.213164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:38.213246 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:38.213639 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:38.213642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:38.213644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:38.213835 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:38.213967 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:38.214274 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 06:29:38.214373 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:29:38.239538 8414 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:38.240602 8414 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:38.246579 8414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c58b966d4040438a97b516da578b5e27: Generated new TSK 0
I20260810 06:29:38.247215 8414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:38.252995 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8269
W20260810 06:29:38.269471 8243 connection.cc:535] client connection to 127.0.0.1:34983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34983): Connection reset by peer (error 104)
2026-08-10T06:29:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1631 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:38 dist-test-slave-45dx krb5kdc[8433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:38 dist-test-slave-45dx krb5kdc[8433](info): set up 2 sockets
Aug 10 06:29:38 dist-test-slave-45dx krb5kdc[8433](info): commencing operation
krb5kdc: starting...
W20260810 06:29:39.511128 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:39 dist-test-slave-45dx krb5kdc[8433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:39.631644 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:39.740919 8453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:39.741168 8453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:39.741225 8453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:39.745168 8453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:39.745218 8453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:39.745237 8453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:39.745254 8453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:39.745270 8453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:39.750319 8453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:39.751252 8453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:39.752205 8453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:29:39.761324 8453 server_base.cc:1034] running on GCE node
W20260810 06:29:39.761264 8461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:39.761255 8460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:39.761250 8467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:39.762101 8453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:39.763203 8453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:39.764380 8453 hybrid_clock.cc:648] HybridClock initialized: now 1786343379764361 us; error 40 us; skew 500 ppm
Aug 10 06:29:39 dist-test-slave-45dx krb5kdc[8433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:39.767146 8453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:39.768016 8453 webserver.cc:460] Webserver started at http://127.0.0.1:38019/ using document root <none> and password file <none>
I20260810 06:29:39.768596 8453 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:39.768662 8453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:39.768832 8453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:39.770833 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "05015e6b92f74e4c8c71bcc385b84dfb"
format_stamp: "Formatted at 2026-08-10 06:29:39 on dist-test-slave-45dx"
server_key: "095f46e3d649b7f9bc98e7dfa23ee648"
server_key_iv: "a944284fa0a53d33a8be460161120e9c"
server_key_version: "encryptionkey@0"
I20260810 06:29:39.771368 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "05015e6b92f74e4c8c71bcc385b84dfb"
format_stamp: "Formatted at 2026-08-10 06:29:39 on dist-test-slave-45dx"
server_key: "095f46e3d649b7f9bc98e7dfa23ee648"
server_key_iv: "a944284fa0a53d33a8be460161120e9c"
server_key_version: "encryptionkey@0"
I20260810 06:29:39.775022 8453 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 06:29:39.777356 8475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:39.778548 8453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:39.778653 8453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "05015e6b92f74e4c8c71bcc385b84dfb"
format_stamp: "Formatted at 2026-08-10 06:29:39 on dist-test-slave-45dx"
server_key: "095f46e3d649b7f9bc98e7dfa23ee648"
server_key_iv: "a944284fa0a53d33a8be460161120e9c"
server_key_version: "encryptionkey@0"
I20260810 06:29:39.778784 8453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:39.806565 8453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:39.809556 8453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:39.809764 8453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:39.816830 8453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41839
I20260810 06:29:39.816854 8573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41839 every 8 connection(s)
I20260810 06:29:39.817746 8453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:39.820477 8574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:39.826092 8574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb: Bootstrap starting.
I20260810 06:29:39.827883 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8453
I20260810 06:29:39.828011 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:39.828275 4388 external_mini_cluster.cc:1428] Setting key 23756cc9fc639dd396b2cdf58814cc62
I20260810 06:29:39.828557 8574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:39.829456 8574 log.cc:824] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:39.831314 8574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb: No bootstrap required, opened a new log
I20260810 06:29:39.835425 8574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41839 } }
I20260810 06:29:39.835657 8574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:39.835711 8574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05015e6b92f74e4c8c71bcc385b84dfb, State: Initialized, Role: FOLLOWER
I20260810 06:29:39.836225 8574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41839 } }
I20260810 06:29:39.836470 8574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:39.836545 8574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:39.836670 8574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:39.837217 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:39.829407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:42477 (local address 127.0.0.1:41839)
0810 06:29:39.829887 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:39.829902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:39.829936 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:29:39.830560 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:39.830576 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:39.830885 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:39.831151 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:39.831529 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:39.832546 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:39.833756 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:39.834214 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:39.834332 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:39.835245 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:39.835250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:39.835261 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:39.835714 (+ 453us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:39.835802 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:39.835836 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:39.835961 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:39.836187 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 06:29:39.836379 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":159,"threads_started":1}
I20260810 06:29:39.837585 8574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41839 } }
I20260810 06:29:39.837931 8574 leader_election.cc:302] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05015e6b92f74e4c8c71bcc385b84dfb; no voters:
I20260810 06:29:39.838256 8574 leader_election.cc:288] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:39.838337 8585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:39.838560 8585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [term 1 LEADER]: Becoming Leader. State: Replica: 05015e6b92f74e4c8c71bcc385b84dfb, State: Running, Role: LEADER
I20260810 06:29:39.838896 8585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41839 } }
I20260810 06:29:39.839452 8574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:39.840485 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:39.840687 8586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41839 } } }
I20260810 06:29:39.840824 8586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:39.841135 8587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05015e6b92f74e4c8c71bcc385b84dfb. Latest consensus state: current_term: 1 leader_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05015e6b92f74e4c8c71bcc385b84dfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41839 } } }
I20260810 06:29:39.841234 8587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb [sys.catalog]: This master's current role is: LEADER
W20260810 06:29:39.844956 8609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:39.845045 8609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:39.845145 8599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:39.845935 8599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:39.847435 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:39.842960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:55567 (local address 127.0.0.1:41839)
0810 06:29:39.843109 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:39.843113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:39.843261 (+ 148us) server_negotiation.cc:407] Connection header received
0810 06:29:39.843349 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:39.843354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:39.843408 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:39.843492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:39.843783 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:39.844544 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:39.845604 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:39.845913 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:39.846026 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:39.846938 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:39.846942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:39.846945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:39.846958 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:39.846984 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:39.846996 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:39.847089 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:39.847251 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 06:29:39.847306 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 06:29:39.851922 8599 catalog_manager.cc:1347] Generated new cluster ID: 5ca25e53947c41d99db73e7599d0616f
I20260810 06:29:39.851984 8599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:39.865204 8599 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:39.866395 8599 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:39.876362 8599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05015e6b92f74e4c8c71bcc385b84dfb: Generated new TSK 0
I20260810 06:29:39.877079 8599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:39.881261 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8453
W20260810 06:29:39.895485 8431 connection.cc:535] client connection to 127.0.0.1:41839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41839): Connection reset by peer (error 104)
2026-08-10T06:29:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:39 dist-test-slave-45dx krb5kdc[8624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:39 dist-test-slave-45dx krb5kdc[8624](info): set up 2 sockets
Aug 10 06:29:39 dist-test-slave-45dx krb5kdc[8624](info): commencing operation
krb5kdc: starting...
W20260810 06:29:41.207510 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:41 dist-test-slave-45dx krb5kdc[8624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 06:29:41.329360 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:41.435940 8650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:41.436246 8650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:41.436348 8650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:41.440163 8650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:41.440251 8650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:41.440290 8650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:41.440369 8650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:41.440419 8650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:41.445920 8650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:41.446942 8650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:41.447925 8650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:41.456069 8666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:41.456048 8668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:41.456065 8670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:41.456372 8650 server_base.cc:1034] running on GCE node
I20260810 06:29:41.456964 8650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:41.457927 8650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:41.459112 8650 hybrid_clock.cc:648] HybridClock initialized: now 1786343381459094 us; error 28 us; skew 500 ppm
Aug 10 06:29:41 dist-test-slave-45dx krb5kdc[8624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:41.461882 8650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 06:29:41.462671 8650 webserver.cc:460] Webserver started at http://127.0.0.1:42537/ using document root <none> and password file <none>
I20260810 06:29:41.463131 8650 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:41.463186 8650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:41.463335 8650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:41.465209 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "45cda0e5a1764cf7af9adf152fb27755"
format_stamp: "Formatted at 2026-08-10 06:29:41 on dist-test-slave-45dx"
server_key: "9d7c5d2ea2969ff5ab09feec3d60279c"
server_key_iv: "3609a88b86c1e351c9d1da9d3ef3e56a"
server_key_version: "encryptionkey@0"
I20260810 06:29:41.465685 8650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "45cda0e5a1764cf7af9adf152fb27755"
format_stamp: "Formatted at 2026-08-10 06:29:41 on dist-test-slave-45dx"
server_key: "9d7c5d2ea2969ff5ab09feec3d60279c"
server_key_iv: "3609a88b86c1e351c9d1da9d3ef3e56a"
server_key_version: "encryptionkey@0"
I20260810 06:29:41.469053 8650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:41.471292 8685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:41.472519 8650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:41.472637 8650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "45cda0e5a1764cf7af9adf152fb27755"
format_stamp: "Formatted at 2026-08-10 06:29:41 on dist-test-slave-45dx"
server_key: "9d7c5d2ea2969ff5ab09feec3d60279c"
server_key_iv: "3609a88b86c1e351c9d1da9d3ef3e56a"
server_key_version: "encryptionkey@0"
I20260810 06:29:41.472729 8650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:41.476902 8650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:41.479497 8650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:41.479682 8650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:41.486547 8650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38893
I20260810 06:29:41.486574 8770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38893 every 8 connection(s)
I20260810 06:29:41.487360 8650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:41.489954 8772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:41.494824 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8650
I20260810 06:29:41.494968 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:41.495260 4388 external_mini_cluster.cc:1428] Setting key b756770488bcb5df8123d4c6174a0db6
I20260810 06:29:41.495455 8772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755: Bootstrap starting.
I20260810 06:29:41.498190 8772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:41.499112 8772 log.cc:824] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:41.501221 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:41.496629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:50113 (local address 127.0.0.1:38893)
0810 06:29:41.497063 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:41.497080 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:41.497120 (+ 40us) server_negotiation.cc:407] Connection header received
0810 06:29:41.497897 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:41.497919 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:41.498264 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:41.498641 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:41.499850 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:41.499857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:41.499875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:41.499891 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:41.499962 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:41.499993 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:41.500124 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:41.500339 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 06:29:41.500537 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260810 06:29:41.501258 8772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755: No bootstrap required, opened a new log
I20260810 06:29:41.504163 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:41.505354 8772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cda0e5a1764cf7af9adf152fb27755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38893 } }
I20260810 06:29:41.505597 8772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:41.505671 8772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cda0e5a1764cf7af9adf152fb27755, State: Initialized, Role: FOLLOWER
I20260810 06:29:41.506114 8772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cda0e5a1764cf7af9adf152fb27755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38893 } }
I20260810 06:29:41.506352 8772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:41.506438 8772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:41.506537 8772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:41.507527 8772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cda0e5a1764cf7af9adf152fb27755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38893 } }
I20260810 06:29:41.507889 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:41.506216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:38981 (local address 127.0.0.1:38893)
0810 06:29:41.506337 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:41.506341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:41.506531 (+ 190us) server_negotiation.cc:407] Connection header received
0810 06:29:41.506689 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:41.506692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:41.506730 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:41.506812 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:41.507515 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:41.507518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:41.507521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:41.507532 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:41.507558 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:41.507575 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:41.507662 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:41.507741 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 06:29:41.507777 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:29:41.507905 8772 leader_election.cc:302] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45cda0e5a1764cf7af9adf152fb27755; no voters:
I20260810 06:29:41.508266 8772 leader_election.cc:288] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:41.508385 8787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:41.508605 8787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [term 1 LEADER]: Becoming Leader. State: Replica: 45cda0e5a1764cf7af9adf152fb27755, State: Running, Role: LEADER
I20260810 06:29:41.508941 8787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cda0e5a1764cf7af9adf152fb27755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38893 } }
I20260810 06:29:41.509531 8772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:41.510000 8787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45cda0e5a1764cf7af9adf152fb27755. Latest consensus state: current_term: 1 leader_uuid: "45cda0e5a1764cf7af9adf152fb27755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cda0e5a1764cf7af9adf152fb27755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38893 } } }
I20260810 06:29:41.510192 8787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:41.510438 8788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45cda0e5a1764cf7af9adf152fb27755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cda0e5a1764cf7af9adf152fb27755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38893 } } }
I20260810 06:29:41.510599 8788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:41.510690 8800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:41.513710 8800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:41.519228 8800 catalog_manager.cc:1347] Generated new cluster ID: 96819c0463034a87bcb82499872a4bad
I20260810 06:29:41.519322 8800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:41.540556 8800 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:41.540858 8800 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:41.549655 8800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45cda0e5a1764cf7af9adf152fb27755: Generated new TSK 0
I20260810 06:29:41.550251 8800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:41.558895 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8650
2026-08-10T06:29:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1678 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:41 dist-test-slave-45dx krb5kdc[8820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:41 dist-test-slave-45dx krb5kdc[8820](info): set up 2 sockets
Aug 10 06:29:41 dist-test-slave-45dx krb5kdc[8820](info): commencing operation
krb5kdc: starting...
W20260810 06:29:42.849536 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:42 dist-test-slave-45dx krb5kdc[8820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:42.983453 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46033
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:46033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:43.096454 8841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:43.096774 8841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:43.096851 8841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:43.100858 8841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:43.100956 8841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:43.100996 8841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:43.101052 8841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:43.101091 8841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:43.106316 8841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:46033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:43.107344 8841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:43.108489 8841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:43.116822 8853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:43.116873 8854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:43.116891 8856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:43.117096 8841 server_base.cc:1034] running on GCE node
I20260810 06:29:43.117683 8841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:43.118683 8841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:43.119845 8841 hybrid_clock.cc:648] HybridClock initialized: now 1786343383119818 us; error 50 us; skew 500 ppm
Aug 10 06:29:43 dist-test-slave-45dx krb5kdc[8820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343383, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:43.122784 8841 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:43.123694 8841 webserver.cc:460] Webserver started at http://127.4.73.62:36241/ using document root <none> and password file <none>
I20260810 06:29:43.124245 8841 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:43.124362 8841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:43.124609 8841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:43.126600 8841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "9744420a21104f899b25c740a290f764"
format_stamp: "Formatted at 2026-08-10 06:29:43 on dist-test-slave-45dx"
server_key: "885765325f69e45bbe789df667fb241c"
server_key_iv: "2497ecb01e74eef13247a7fd545f08cd"
server_key_version: "encryptionkey@0"
I20260810 06:29:43.127171 8841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "9744420a21104f899b25c740a290f764"
format_stamp: "Formatted at 2026-08-10 06:29:43 on dist-test-slave-45dx"
server_key: "885765325f69e45bbe789df667fb241c"
server_key_iv: "2497ecb01e74eef13247a7fd545f08cd"
server_key_version: "encryptionkey@0"
I20260810 06:29:43.130857 8841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 06:29:43.134886 8871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:43.136186 8841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:29:43.136313 8841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "9744420a21104f899b25c740a290f764"
format_stamp: "Formatted at 2026-08-10 06:29:43 on dist-test-slave-45dx"
server_key: "885765325f69e45bbe789df667fb241c"
server_key_iv: "2497ecb01e74eef13247a7fd545f08cd"
server_key_version: "encryptionkey@0"
I20260810 06:29:43.136467 8841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:43.160714 8841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:43.164004 8841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:43.164213 8841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:43.171411 8841 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:46033
I20260810 06:29:43.171429 8955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:46033 every 8 connection(s)
I20260810 06:29:43.172360 8841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:43.175052 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:43.180629 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764: Bootstrap starting.
I20260810 06:29:43.180652 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 8841
I20260810 06:29:43.180823 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:43.181107 4388 external_mini_cluster.cc:1428] Setting key a27d4f187543ce719452b7dc4dd10e36
I20260810 06:29:43.183270 8956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:43.184033 8956 log.cc:824] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:43.186178 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764: No bootstrap required, opened a new log
Aug 10 06:29:43 dist-test-slave-45dx krb5kdc[8820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:43.189950 8956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9744420a21104f899b25c740a290f764" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46033 } }
I20260810 06:29:43.190217 8956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:43.190289 8956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9744420a21104f899b25c740a290f764, State: Initialized, Role: FOLLOWER
I20260810 06:29:43.190752 8956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9744420a21104f899b25c740a290f764" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46033 } }
I20260810 06:29:43.190917 8956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:43.191004 8956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:43.191125 8956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:43.192008 8956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9744420a21104f899b25c740a290f764" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46033 } }
I20260810 06:29:43.192390 8956 leader_election.cc:302] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9744420a21104f899b25c740a290f764; no voters:
I20260810 06:29:43.192756 8956 leader_election.cc:288] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:43.192818 8967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:43.193035 8967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [term 1 LEADER]: Becoming Leader. State: Replica: 9744420a21104f899b25c740a290f764, State: Running, Role: LEADER
I20260810 06:29:43.193388 8967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9744420a21104f899b25c740a290f764" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46033 } }
I20260810 06:29:43.194057 8956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:43.195082 8971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9744420a21104f899b25c740a290f764. Latest consensus state: current_term: 1 leader_uuid: "9744420a21104f899b25c740a290f764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9744420a21104f899b25c740a290f764" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46033 } } }
I20260810 06:29:43.195237 8971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:43.195577 8968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9744420a21104f899b25c740a290f764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9744420a21104f899b25c740a290f764" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46033 } } }
I20260810 06:29:43.195736 8968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:43.197047 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:43.182473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53679 (local address 127.4.73.62:46033)
0810 06:29:43.182927 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:43.182941 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:43.182976 (+ 35us) server_negotiation.cc:407] Connection header received
0810 06:29:43.183659 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:43.183677 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:43.184013 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:43.184392 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:43.185225 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:43.186217 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:43.187130 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:43.187661 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:43.187844 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:43.190826 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:43.190831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:43.190843 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:43.190870 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:43.193072 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:43.193699 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:43.193703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:43.193708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:43.193816 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:43.194174 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:43.194177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:43.194179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:43.194711 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:43.195651 (+ 940us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:43.195820 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 06:29:43.196069 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
W20260810 06:29:43.199374 9001 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:43.199465 9001 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:43.199564 8984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:43.200188 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:43.200779 8984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:43.206642 8984 catalog_manager.cc:1347] Generated new cluster ID: a440b1bdfd744fa6b8b5f346d1b3d13e
I20260810 06:29:43.206707 8984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:43.211216 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:43.202491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35807 (local address 127.4.73.62:46033)
0810 06:29:43.202603 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:43.202607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:43.202940 (+ 333us) server_negotiation.cc:407] Connection header received
0810 06:29:43.203077 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:43.203081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:43.203128 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:43.203216 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:43.203857 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:43.204459 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:43.205326 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:43.205696 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:43.205809 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:43.207605 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:43.207608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:43.207611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:43.207634 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:43.209385 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:43.209930 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:43.209933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:43.209935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:43.210013 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:43.210364 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:43.210367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:43.210368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:43.210549 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:43.210701 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:43.210985 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 06:29:43.211076 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 06:29:43.220002 8984 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:43.221185 8984 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:43.226856 8984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9744420a21104f899b25c740a290f764: Generated new TSK 0
I20260810 06:29:43.227553 8984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:43.232257 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 8841
W20260810 06:29:43.248715 8817 connection.cc:535] client connection to 127.4.73.62:46033 recv error: Network error: failed to read from TLS socket (remote: 127.4.73.62:46033): Connection reset by peer (error 104)
2026-08-10T06:29:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1675 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:43 dist-test-slave-45dx krb5kdc[9018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:43 dist-test-slave-45dx krb5kdc[9018](info): set up 2 sockets
Aug 10 06:29:43 dist-test-slave-45dx krb5kdc[9018](info): commencing operation
krb5kdc: starting...
W20260810 06:29:44.498538 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:44 dist-test-slave-45dx krb5kdc[9018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:44.619102 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:39709
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:44.729882 9036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:44.730187 9036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:44.730268 9036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:44.734319 9036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:44.734416 9036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:44.734455 9036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:44.734496 9036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:44.734529 9036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:44.739710 9036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:39709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:44.740937 9036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:44.742018 9036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:44.750931 9043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:44.751026 9036 server_base.cc:1034] running on GCE node
W20260810 06:29:44.750965 9052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:44.750931 9050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:44.751746 9036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:44.752868 9036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:44.754029 9036 hybrid_clock.cc:648] HybridClock initialized: now 1786343384754018 us; error 25 us; skew 500 ppm
Aug 10 06:29:44 dist-test-slave-45dx krb5kdc[9018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343384, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:44.756625 9036 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:44.757472 9036 webserver.cc:460] Webserver started at http://127.4.73.62:41239/ using document root <none> and password file <none>
I20260810 06:29:44.758037 9036 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:44.758127 9036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:44.758351 9036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:44.760389 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "60854e242e3b48da951131172a04b3cb"
format_stamp: "Formatted at 2026-08-10 06:29:44 on dist-test-slave-45dx"
server_key: "503d6956fa3c4f9664b682f633a911ad"
server_key_iv: "d36465896c69d8cfadc6eb1c73a88042"
server_key_version: "encryptionkey@0"
I20260810 06:29:44.760974 9036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "60854e242e3b48da951131172a04b3cb"
format_stamp: "Formatted at 2026-08-10 06:29:44 on dist-test-slave-45dx"
server_key: "503d6956fa3c4f9664b682f633a911ad"
server_key_iv: "d36465896c69d8cfadc6eb1c73a88042"
server_key_version: "encryptionkey@0"
I20260810 06:29:44.764550 9036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 06:29:44.767099 9069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:44.768553 9036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:29:44.768690 9036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "60854e242e3b48da951131172a04b3cb"
format_stamp: "Formatted at 2026-08-10 06:29:44 on dist-test-slave-45dx"
server_key: "503d6956fa3c4f9664b682f633a911ad"
server_key_iv: "d36465896c69d8cfadc6eb1c73a88042"
server_key_version: "encryptionkey@0"
I20260810 06:29:44.768792 9036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:44.811328 9036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:44.814985 9036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:44.815147 9036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:44.822256 9036 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:39709
I20260810 06:29:44.822274 9161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:39709 every 8 connection(s)
I20260810 06:29:44.823254 9036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:44.825889 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9036
I20260810 06:29:44.826040 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:44.826227 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:44.826390 4388 external_mini_cluster.cc:1428] Setting key 7a17437cd01665bc4e9ca8dc19833b87
I20260810 06:29:44.833619 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb: Bootstrap starting.
I20260810 06:29:44.836220 9162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:44.836666 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:44.827762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48491 (local address 127.4.73.62:39709)
0810 06:29:44.828215 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:44.828234 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:44.828414 (+ 180us) server_negotiation.cc:407] Connection header received
0810 06:29:44.829182 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:44.829204 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:44.829646 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:44.830034 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:44.830445 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:44.831907 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:44.832848 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:44.833527 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:44.833691 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:44.834600 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:44.834607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:44.834624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:44.835125 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:44.835190 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:44.835219 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:44.835348 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:44.835508 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 06:29:44.835635 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
I20260810 06:29:44.837145 9162 log.cc:824] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:44.839308 9162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb: No bootstrap required, opened a new log
I20260810 06:29:44.839654 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:44.843547 9162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60854e242e3b48da951131172a04b3cb" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39709 } }
I20260810 06:29:44.843860 9162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:44.843918 9162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60854e242e3b48da951131172a04b3cb, State: Initialized, Role: FOLLOWER
I20260810 06:29:44.844440 9162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60854e242e3b48da951131172a04b3cb" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39709 } }
I20260810 06:29:44.844619 9162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:44.844750 9162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:44.844871 9162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:44.845819 9162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60854e242e3b48da951131172a04b3cb" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39709 } }
I20260810 06:29:44.846115 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:44.841817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52511 (local address 127.4.73.62:39709)
0810 06:29:44.841941 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:44.841945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:44.842090 (+ 145us) server_negotiation.cc:407] Connection header received
0810 06:29:44.842202 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:44.842205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:44.842241 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:44.842308 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:44.842612 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:44.843094 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:44.844229 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:44.844595 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:44.844695 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:44.845642 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:44.845646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:44.845649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:44.845661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:44.845695 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:44.845706 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:44.845803 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:44.845926 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 06:29:44.845958 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 06:29:44.846199 9162 leader_election.cc:302] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60854e242e3b48da951131172a04b3cb; no voters:
I20260810 06:29:44.846545 9162 leader_election.cc:288] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:44.846628 9176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:44.846825 9176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [term 1 LEADER]: Becoming Leader. State: Replica: 60854e242e3b48da951131172a04b3cb, State: Running, Role: LEADER
I20260810 06:29:44.847229 9176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60854e242e3b48da951131172a04b3cb" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39709 } }
I20260810 06:29:44.847826 9162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:44.848948 9179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60854e242e3b48da951131172a04b3cb. Latest consensus state: current_term: 1 leader_uuid: "60854e242e3b48da951131172a04b3cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60854e242e3b48da951131172a04b3cb" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39709 } } }
I20260810 06:29:44.849077 9179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:44.849457 9190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:44.849517 9177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60854e242e3b48da951131172a04b3cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60854e242e3b48da951131172a04b3cb" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 39709 } } }
I20260810 06:29:44.850220 9177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:44.852735 9190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:44.858307 9190 catalog_manager.cc:1347] Generated new cluster ID: c0cc5aaead62471fab2aba0df3e3d3f7
I20260810 06:29:44.858372 9190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:44.864537 9190 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:44.865410 9190 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:44.871083 9190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60854e242e3b48da951131172a04b3cb: Generated new TSK 0
I20260810 06:29:44.871825 9190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:44.879106 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9036
W20260810 06:29:44.896154 9014 connection.cc:535] client connection to 127.4.73.62:39709 recv error: Network error: failed to read from TLS socket (remote: 127.4.73.62:39709): Connection reset by peer (error 104)
2026-08-10T06:29:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1647 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:44 dist-test-slave-45dx krb5kdc[9209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:44 dist-test-slave-45dx krb5kdc[9209](info): set up 2 sockets
Aug 10 06:29:44 dist-test-slave-45dx krb5kdc[9209](info): commencing operation
krb5kdc: starting...
W20260810 06:29:46.163058 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:46 dist-test-slave-45dx krb5kdc[9209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:46.313498 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46005
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:46005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:46.424012 9234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:46.424242 9234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:46.424309 9234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:46.428059 9234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:46.428115 9234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:46.428133 9234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:46.428149 9234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:46.428164 9234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:46.433223 9234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:46005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:46005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:46.434146 9234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:46.435107 9234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:46.443727 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:46.443720 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:46.443720 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:46.443830 9234 server_base.cc:1034] running on GCE node
I20260810 06:29:46.444603 9234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:46.445637 9234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:46.446833 9234 hybrid_clock.cc:648] HybridClock initialized: now 1786343386446806 us; error 30 us; skew 500 ppm
Aug 10 06:29:46 dist-test-slave-45dx krb5kdc[9209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:46.453804 9234 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:46.454613 9234 webserver.cc:460] Webserver started at http://127.4.73.62:46163/ using document root <none> and password file <none>
I20260810 06:29:46.455142 9234 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:46.455204 9234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:46.455363 9234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:46.457412 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "f91013d329564923a3cfb592797d65cf"
format_stamp: "Formatted at 2026-08-10 06:29:46 on dist-test-slave-45dx"
server_key: "99095222c2176387718abcbd8d1d7a8d"
server_key_iv: "74b1de4a3e6fe4bae648a88aba98ea68"
server_key_version: "encryptionkey@0"
I20260810 06:29:46.457940 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "f91013d329564923a3cfb592797d65cf"
format_stamp: "Formatted at 2026-08-10 06:29:46 on dist-test-slave-45dx"
server_key: "99095222c2176387718abcbd8d1d7a8d"
server_key_iv: "74b1de4a3e6fe4bae648a88aba98ea68"
server_key_version: "encryptionkey@0"
I20260810 06:29:46.461602 9234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:46.463857 9260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:46.465085 9234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:46.465226 9234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "f91013d329564923a3cfb592797d65cf"
format_stamp: "Formatted at 2026-08-10 06:29:46 on dist-test-slave-45dx"
server_key: "99095222c2176387718abcbd8d1d7a8d"
server_key_iv: "74b1de4a3e6fe4bae648a88aba98ea68"
server_key_version: "encryptionkey@0"
I20260810 06:29:46.465322 9234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:46.469851 9234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:46.473048 9234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:46.473239 9234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:46.480005 9234 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:46005
I20260810 06:29:46.480020 9356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:46005 every 8 connection(s)
I20260810 06:29:46.480903 9234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:46.483515 9357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:46.488695 9357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf: Bootstrap starting.
I20260810 06:29:46.489643 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9234
I20260810 06:29:46.489817 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:46.490134 4388 external_mini_cluster.cc:1428] Setting key b3237808e83d49ad5ba09697a73750a7
I20260810 06:29:46.491496 9357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:46.492197 9357 log.cc:824] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:46.494084 9357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf: No bootstrap required, opened a new log
I20260810 06:29:46.494755 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:46.491174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35733 (local address 127.4.73.62:46005)
0810 06:29:46.491576 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:46.491596 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:46.491638 (+ 42us) server_negotiation.cc:407] Connection header received
0810 06:29:46.492215 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:46.492230 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:46.492562 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:46.492884 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:46.493609 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:46.493614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:46.493624 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:46.493634 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:46.493683 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:46.493701 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:46.493806 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:46.493912 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 06:29:46.494049 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":126,"threads_started":1}
I20260810 06:29:46.497323 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:46.497630 9357 raft_consensus.cc:348] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91013d329564923a3cfb592797d65cf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46005 } }
I20260810 06:29:46.497822 9357 raft_consensus.cc:374] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:46.497929 9357 raft_consensus.cc:729] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f91013d329564923a3cfb592797d65cf, State: Initialized, Role: FOLLOWER
I20260810 06:29:46.498389 9357 consensus_queue.cc:260] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91013d329564923a3cfb592797d65cf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46005 } }
I20260810 06:29:46.498535 9357 raft_consensus.cc:388] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:46.498607 9357 raft_consensus.cc:482] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:46.498715 9357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:46.499691 9357 raft_consensus.cc:504] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91013d329564923a3cfb592797d65cf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46005 } }
I20260810 06:29:46.500077 9357 leader_election.cc:302] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f91013d329564923a3cfb592797d65cf; no voters:
I20260810 06:29:46.500433 9357 leader_election.cc:288] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:46.500583 9371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:46.500808 9371 raft_consensus.cc:686] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [term 1 LEADER]: Becoming Leader. State: Replica: f91013d329564923a3cfb592797d65cf, State: Running, Role: LEADER
I20260810 06:29:46.501129 9371 consensus_queue.cc:237] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91013d329564923a3cfb592797d65cf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46005 } }
I20260810 06:29:46.501495 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:46.499470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38663 (local address 127.4.73.62:46005)
0810 06:29:46.499582 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:46.499586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:46.499779 (+ 193us) server_negotiation.cc:407] Connection header received
0810 06:29:46.499920 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:46.499923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:46.499957 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:46.500031 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:46.500785 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:46.500789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:46.500791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:46.500802 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:46.500829 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:46.500841 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:46.501300 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:46.501338 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 06:29:46.501374 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:29:46.501763 9357 sys_catalog.cc:564] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:46.503280 9372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f91013d329564923a3cfb592797d65cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91013d329564923a3cfb592797d65cf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46005 } } }
I20260810 06:29:46.503409 9372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:46.503628 9375 sys_catalog.cc:455] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f91013d329564923a3cfb592797d65cf. Latest consensus state: current_term: 1 leader_uuid: "f91013d329564923a3cfb592797d65cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f91013d329564923a3cfb592797d65cf" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 46005 } } }
I20260810 06:29:46.503712 9375 sys_catalog.cc:458] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:46.503911 9389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:46.506923 9389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:46.512390 9389 catalog_manager.cc:1347] Generated new cluster ID: a1419469a4764aa4bc0cec0608acd0fd
I20260810 06:29:46.512452 9389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:46.525179 9389 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:46.525470 9389 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:46.537534 9389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f91013d329564923a3cfb592797d65cf: Generated new TSK 0
I20260810 06:29:46.538312 9389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:46.544023 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9234
2026-08-10T06:29:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1664 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:46 dist-test-slave-45dx krb5kdc[9407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:46 dist-test-slave-45dx krb5kdc[9407](info): set up 2 sockets
Aug 10 06:29:46 dist-test-slave-45dx krb5kdc[9407](info): commencing operation
krb5kdc: starting...
W20260810 06:29:47.791275 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:47 dist-test-slave-45dx krb5kdc[9407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:47.923436 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43355
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:44213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:43355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:48.041050 9431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:48.041296 9431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:48.041334 9431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:48.045192 9431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:48.045258 9431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:48.045277 9431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:48.045293 9431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:48.045308 9431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:48.050225 9431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:44213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:43355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:43355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:48.051163 9431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:48.052069 9431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:48.060454 9446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:48.060436 9443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:48.060418 9442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:48.060765 9431 server_base.cc:1034] running on GCE node
I20260810 06:29:48.061455 9431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:48.062494 9431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:48.063709 9431 hybrid_clock.cc:648] HybridClock initialized: now 1786343388063677 us; error 39 us; skew 500 ppm
Aug 10 06:29:48 dist-test-slave-45dx krb5kdc[9407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:48.066967 9431 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:48.067906 9431 webserver.cc:460] Webserver started at http://127.4.73.62:34199/ using document root <none> and password file <none>
I20260810 06:29:48.068547 9431 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:48.068614 9431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:48.068907 9431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:48.071085 9431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "72cbc84038ce41ed9f61e5bbce99889b"
format_stamp: "Formatted at 2026-08-10 06:29:48 on dist-test-slave-45dx"
server_key: "b795cb6e46ca5e15c596425cdd469b14"
server_key_iv: "97892a54be8d5b5583c170fe4d5ef50b"
server_key_version: "encryptionkey@0"
I20260810 06:29:48.071686 9431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "72cbc84038ce41ed9f61e5bbce99889b"
format_stamp: "Formatted at 2026-08-10 06:29:48 on dist-test-slave-45dx"
server_key: "b795cb6e46ca5e15c596425cdd469b14"
server_key_iv: "97892a54be8d5b5583c170fe4d5ef50b"
server_key_version: "encryptionkey@0"
I20260810 06:29:48.075507 9431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 06:29:48.078032 9454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:48.079232 9431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:48.079380 9431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "72cbc84038ce41ed9f61e5bbce99889b"
format_stamp: "Formatted at 2026-08-10 06:29:48 on dist-test-slave-45dx"
server_key: "b795cb6e46ca5e15c596425cdd469b14"
server_key_iv: "97892a54be8d5b5583c170fe4d5ef50b"
server_key_version: "encryptionkey@0"
I20260810 06:29:48.079481 9431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:48.095782 9431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:48.098953 9431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:48.099128 9431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:48.106125 9431 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:43355
I20260810 06:29:48.106151 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:43355 every 8 connection(s)
I20260810 06:29:48.107021 9431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:48.109601 9547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:48.111358 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9431
I20260810 06:29:48.111505 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:48.111784 4388 external_mini_cluster.cc:1428] Setting key 9dbfe1446ce0743fefbc6876f76cb13e
I20260810 06:29:48.115597 9547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b: Bootstrap starting.
I20260810 06:29:48.117749 9547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:48.118434 9547 log.cc:824] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b: Log is configured to *not* fsync() on all Append() calls
Aug 10 06:29:48 dist-test-slave-45dx krb5kdc[9407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:48.120452 9547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b: No bootstrap required, opened a new log
I20260810 06:29:48.123574 9547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbc84038ce41ed9f61e5bbce99889b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43355 } }
I20260810 06:29:48.123843 9547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:48.123974 9547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cbc84038ce41ed9f61e5bbce99889b, State: Initialized, Role: FOLLOWER
I20260810 06:29:48.124500 9547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbc84038ce41ed9f61e5bbce99889b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43355 } }
I20260810 06:29:48.124665 9547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:48.124742 9547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:48.124851 9547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:48.125916 9547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbc84038ce41ed9f61e5bbce99889b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43355 } }
I20260810 06:29:48.126257 9547 leader_election.cc:302] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72cbc84038ce41ed9f61e5bbce99889b; no voters:
I20260810 06:29:48.126464 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:48.113132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35409 (local address 127.4.73.62:43355)
0810 06:29:48.113544 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:48.113558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:48.113591 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:29:48.114198 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:48.114214 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:48.114525 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:48.114837 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:48.115628 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:48.116704 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:48.117562 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:48.118074 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:48.118245 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:48.121006 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:48.121011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:48.121023 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:48.121048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:48.123207 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:48.123695 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:48.123699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:48.123704 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:48.123808 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:48.124136 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:48.124139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:48.124141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:48.124676 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:48.124902 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:48.125209 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 06:29:48.125487 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":126,"threads_started":1}
I20260810 06:29:48.126582 9547 leader_election.cc:288] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:48.126705 9554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:48.126953 9554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [term 1 LEADER]: Becoming Leader. State: Replica: 72cbc84038ce41ed9f61e5bbce99889b, State: Running, Role: LEADER
I20260810 06:29:48.127292 9554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbc84038ce41ed9f61e5bbce99889b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43355 } }
I20260810 06:29:48.127708 9547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:48.128482 9554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72cbc84038ce41ed9f61e5bbce99889b. Latest consensus state: current_term: 1 leader_uuid: "72cbc84038ce41ed9f61e5bbce99889b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbc84038ce41ed9f61e5bbce99889b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43355 } } }
I20260810 06:29:48.128597 9554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:48.129276 9555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72cbc84038ce41ed9f61e5bbce99889b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72cbc84038ce41ed9f61e5bbce99889b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 43355 } } }
I20260810 06:29:48.129369 9555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:48.129647 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 06:29:48.132294 9579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:48.133033 9579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:48.133133 9571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:48.133934 9571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:48.139315 9571 catalog_manager.cc:1347] Generated new cluster ID: 519ce6caa84c428f955de159d6a1792d
I20260810 06:29:48.139382 9571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:48.140509 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:48.131903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46283 (local address 127.4.73.62:43355)
0810 06:29:48.132013 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:48.132016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:48.132333 (+ 317us) server_negotiation.cc:407] Connection header received
0810 06:29:48.132428 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:48.132432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:48.132478 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:48.132558 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:48.133305 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:48.134021 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:48.135123 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:48.135439 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:48.135551 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:48.136944 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:48.136948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:48.136950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:48.136979 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:48.138814 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:48.139249 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:48.139254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:48.139256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:48.139331 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:48.139672 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:48.139675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:48.139676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:48.139838 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:48.139990 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:48.140218 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 06:29:48.140364 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:29:48.156287 9571 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:48.157272 9571 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:48.163069 9571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72cbc84038ce41ed9f61e5bbce99889b: Generated new TSK 0
I20260810 06:29:48.163604 9571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:48.168646 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9431
2026-08-10T06:29:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1622 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:48 dist-test-slave-45dx krb5kdc[9601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:48 dist-test-slave-45dx krb5kdc[9601](info): set up 2 sockets
Aug 10 06:29:48 dist-test-slave-45dx krb5kdc[9601](info): commencing operation
krb5kdc: starting...
W20260810 06:29:49.421135 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:49 dist-test-slave-45dx krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:49.548941 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:40585
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:40585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:49.658078 9624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:49.658334 9624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:49.658372 9624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:49.662211 9624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:49.662259 9624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:49.662278 9624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:49.662294 9624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:49.662322 9624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:49.667160 9624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:40585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:49.668044 9624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:49.668980 9624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:49.677163 9635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:49.677146 9636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:49.677201 9624 server_base.cc:1034] running on GCE node
W20260810 06:29:49.677147 9639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:49.677942 9624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:49.678952 9624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:49.680119 9624 hybrid_clock.cc:648] HybridClock initialized: now 1786343389680101 us; error 30 us; skew 500 ppm
Aug 10 06:29:49 dist-test-slave-45dx krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343389, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:49.683015 9624 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:49.683843 9624 webserver.cc:460] Webserver started at http://127.4.73.62:36525/ using document root <none> and password file <none>
I20260810 06:29:49.684489 9624 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:49.684556 9624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:49.684715 9624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:49.686635 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "f562670726094ed891a20cf2ae8ff61e"
format_stamp: "Formatted at 2026-08-10 06:29:49 on dist-test-slave-45dx"
server_key: "74184d42854438f5c6125879a67a8cde"
server_key_iv: "2fb7408f29d83b1cf6559eaf3fecbd73"
server_key_version: "encryptionkey@0"
I20260810 06:29:49.687142 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "f562670726094ed891a20cf2ae8ff61e"
format_stamp: "Formatted at 2026-08-10 06:29:49 on dist-test-slave-45dx"
server_key: "74184d42854438f5c6125879a67a8cde"
server_key_iv: "2fb7408f29d83b1cf6559eaf3fecbd73"
server_key_version: "encryptionkey@0"
I20260810 06:29:49.690659 9624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 06:29:49.693048 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:49.694312 9624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 06:29:49.694410 9624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "f562670726094ed891a20cf2ae8ff61e"
format_stamp: "Formatted at 2026-08-10 06:29:49 on dist-test-slave-45dx"
server_key: "74184d42854438f5c6125879a67a8cde"
server_key_iv: "2fb7408f29d83b1cf6559eaf3fecbd73"
server_key_version: "encryptionkey@0"
I20260810 06:29:49.694510 9624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:49.717237 9624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:49.720567 9624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:49.720726 9624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:49.727696 9624 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:40585
I20260810 06:29:49.727789 9729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:40585 every 8 connection(s)
I20260810 06:29:49.728688 9624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:49.731431 9733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:49.734777 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9624
I20260810 06:29:49.734933 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:49.735224 4388 external_mini_cluster.cc:1428] Setting key 5e326768af6e12dfec3872538c50a6f4
I20260810 06:29:49.737190 9733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e: Bootstrap starting.
I20260810 06:29:49.739429 9733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:49.740371 9733 log.cc:824] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:49.742539 9733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e: No bootstrap required, opened a new log
I20260810 06:29:49.744040 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:49.736540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49575 (local address 127.4.73.62:40585)
0810 06:29:49.736932 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:49.736947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:49.736983 (+ 36us) server_negotiation.cc:407] Connection header received
0810 06:29:49.737615 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:49.737632 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:49.737943 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:49.738226 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:49.738578 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:49.739520 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:49.740798 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:49.741400 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:49.741523 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:49.742406 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:49.742411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:49.742423 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:49.742839 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:49.742892 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:49.742915 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:49.743004 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:49.743172 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 06:29:49.743382 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260810 06:29:49.746404 9733 raft_consensus.cc:348] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562670726094ed891a20cf2ae8ff61e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40585 } }
I20260810 06:29:49.746663 9733 raft_consensus.cc:374] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:49.746726 9733 raft_consensus.cc:729] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f562670726094ed891a20cf2ae8ff61e, State: Initialized, Role: FOLLOWER
I20260810 06:29:49.746977 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:49.747179 9733 consensus_queue.cc:260] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562670726094ed891a20cf2ae8ff61e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40585 } }
I20260810 06:29:49.747326 9733 raft_consensus.cc:388] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:49.747382 9733 raft_consensus.cc:482] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:49.747486 9733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:49.748569 9733 raft_consensus.cc:504] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562670726094ed891a20cf2ae8ff61e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40585 } }
I20260810 06:29:49.748905 9733 leader_election.cc:302] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f562670726094ed891a20cf2ae8ff61e; no voters:
I20260810 06:29:49.749249 9733 leader_election.cc:288] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:49.749315 9749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:49.749547 9749 raft_consensus.cc:686] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [term 1 LEADER]: Becoming Leader. State: Replica: f562670726094ed891a20cf2ae8ff61e, State: Running, Role: LEADER
I20260810 06:29:49.749889 9749 consensus_queue.cc:237] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562670726094ed891a20cf2ae8ff61e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40585 } }
I20260810 06:29:49.750499 9733 sys_catalog.cc:564] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:49.751029 9749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f562670726094ed891a20cf2ae8ff61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562670726094ed891a20cf2ae8ff61e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40585 } } }
I20260810 06:29:49.751205 9749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:49.751471 9753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f562670726094ed891a20cf2ae8ff61e. Latest consensus state: current_term: 1 leader_uuid: "f562670726094ed891a20cf2ae8ff61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f562670726094ed891a20cf2ae8ff61e" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 40585 } } }
I20260810 06:29:49.751636 9753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:49.751649 9767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:49.753783 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:49.749159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45181 (local address 127.4.73.62:40585)
0810 06:29:49.749448 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:49.749454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:49.749488 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:29:49.749647 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:49.749650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:49.749687 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:49.749731 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:49.750162 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:49.750747 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:49.751936 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:49.752255 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:49.752370 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:49.753355 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:49.753359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:49.753361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:49.753373 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:49.753399 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:49.753411 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:49.753452 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:49.753612 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 06:29:49.753663 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260810 06:29:49.754873 9767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:49.760423 9767 catalog_manager.cc:1347] Generated new cluster ID: b9cdf138fda742ad96eee6a9892a3f71
I20260810 06:29:49.760550 9767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:49.798244 9767 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:49.799335 9767 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:49.809384 9767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f562670726094ed891a20cf2ae8ff61e: Generated new TSK 0
I20260810 06:29:49.810048 9767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:49.820178 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9624
2026-08-10T06:29:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1653 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:49 dist-test-slave-45dx krb5kdc[9795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:49 dist-test-slave-45dx krb5kdc[9795](info): set up 2 sockets
Aug 10 06:29:49 dist-test-slave-45dx krb5kdc[9795](info): commencing operation
krb5kdc: starting...
W20260810 06:29:51.097899 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:51 dist-test-slave-45dx krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:51.227397 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34425
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:43099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:51.338440 9819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:51.338707 9819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:51.338776 9819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:51.342746 9819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:51.342836 9819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:51.342876 9819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:51.342947 9819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:51.342984 9819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:51.347834 9819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:43099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:51.348774 9819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:51.349682 9819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:51.357990 9826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:51.357959 9828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:51.357972 9825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:51.358225 9819 server_base.cc:1034] running on GCE node
I20260810 06:29:51.358868 9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:51.359862 9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:51.361057 9819 hybrid_clock.cc:648] HybridClock initialized: now 1786343391361041 us; error 32 us; skew 500 ppm
Aug 10 06:29:51 dist-test-slave-45dx krb5kdc[9795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343391, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:51.363862 9819 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:51.364809 9819 webserver.cc:460] Webserver started at http://127.4.73.62:46393/ using document root <none> and password file <none>
I20260810 06:29:51.365357 9819 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:51.365451 9819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:51.365661 9819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:51.367619 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "17f4a76ba6a249829e250a35c983c316"
format_stamp: "Formatted at 2026-08-10 06:29:51 on dist-test-slave-45dx"
server_key: "191ff9439c842eb97628ed6ee394a99e"
server_key_iv: "060e19bbac64d16829a8ff0067f9ed75"
server_key_version: "encryptionkey@0"
I20260810 06:29:51.368196 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "17f4a76ba6a249829e250a35c983c316"
format_stamp: "Formatted at 2026-08-10 06:29:51 on dist-test-slave-45dx"
server_key: "191ff9439c842eb97628ed6ee394a99e"
server_key_iv: "060e19bbac64d16829a8ff0067f9ed75"
server_key_version: "encryptionkey@0"
I20260810 06:29:51.371760 9819 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:29:51.374161 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:51.375394 9819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:51.375541 9819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "17f4a76ba6a249829e250a35c983c316"
format_stamp: "Formatted at 2026-08-10 06:29:51 on dist-test-slave-45dx"
server_key: "191ff9439c842eb97628ed6ee394a99e"
server_key_iv: "060e19bbac64d16829a8ff0067f9ed75"
server_key_version: "encryptionkey@0"
I20260810 06:29:51.375662 9819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:51.380096 9819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:51.383410 9819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:51.383623 9819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:51.390834 9819 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:34425
I20260810 06:29:51.390908 9929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:34425 every 8 connection(s)
I20260810 06:29:51.391747 9819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:51.393412 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 9819
I20260810 06:29:51.393549 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:51.393857 4388 external_mini_cluster.cc:1428] Setting key 3335d369b6ae04935c02c744c9be83b4
I20260810 06:29:51.394989 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:51.398746 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:51.395210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40115 (local address 127.4.73.62:34425)
0810 06:29:51.395643 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:51.395657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:51.395691 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:29:51.396354 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:51.396383 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:51.396639 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:51.396950 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:51.397653 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:51.397658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:51.397678 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:51.397687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:51.397732 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:51.397749 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:51.397897 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:51.397964 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 06:29:51.398102 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260810 06:29:51.401139 9930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316: Bootstrap starting.
I20260810 06:29:51.401646 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:51.403399 9930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:51.404239 9930 log.cc:824] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:51.405397 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:51.403672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49731 (local address 127.4.73.62:34425)
0810 06:29:51.403784 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:51.403789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:51.403976 (+ 187us) server_negotiation.cc:407] Connection header received
0810 06:29:51.404091 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:51.404094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:51.404143 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:51.404234 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:51.405001 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:51.405004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:51.405006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 06:29:51.405016 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:51.405062 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 06:29:51.405077 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:51.405179 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:51.405260 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 06:29:51.405295 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 06:29:51.406507 9930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316: No bootstrap required, opened a new log
I20260810 06:29:51.410041 9930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4a76ba6a249829e250a35c983c316" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34425 } }
I20260810 06:29:51.410318 9930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 06:29:51.410391 9930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f4a76ba6a249829e250a35c983c316, State: Initialized, Role: FOLLOWER
I20260810 06:29:51.410758 9930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4a76ba6a249829e250a35c983c316" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34425 } }
I20260810 06:29:51.410876 9930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 06:29:51.410918 9930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 06:29:51.410985 9930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:51.411798 9930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4a76ba6a249829e250a35c983c316" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34425 } }
I20260810 06:29:51.412079 9930 leader_election.cc:302] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f4a76ba6a249829e250a35c983c316; no voters:
I20260810 06:29:51.412405 9930 leader_election.cc:288] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 06:29:51.412482 9944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:51.412770 9944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [term 1 LEADER]: Becoming Leader. State: Replica: 17f4a76ba6a249829e250a35c983c316, State: Running, Role: LEADER
I20260810 06:29:51.413250 9944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4a76ba6a249829e250a35c983c316" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34425 } }
I20260810 06:29:51.413616 9930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:51.414848 9948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f4a76ba6a249829e250a35c983c316. Latest consensus state: current_term: 1 leader_uuid: "17f4a76ba6a249829e250a35c983c316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4a76ba6a249829e250a35c983c316" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34425 } } }
I20260810 06:29:51.414991 9948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:51.415347 9947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f4a76ba6a249829e250a35c983c316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f4a76ba6a249829e250a35c983c316" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 34425 } } }
I20260810 06:29:51.415439 9947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316 [sys.catalog]: This master's current role is: LEADER
W20260810 06:29:51.418759 9969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:51.418843 9969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:51.418953 9961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:51.419687 9961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:51.424729 9961 catalog_manager.cc:1347] Generated new cluster ID: a79a1379955b49e1a89881e9f8c6f700
I20260810 06:29:51.424795 9961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:51.441773 9961 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:51.442103 9961 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:51.463162 9961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f4a76ba6a249829e250a35c983c316: Generated new TSK 0
I20260810 06:29:51.463994 9961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 06:29:51.480635 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 9819
2026-08-10T06:29:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1655 ms)
[----------] 18 tests from AuthTokenIssuingTest (29433 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:51 dist-test-slave-45dx krb5kdc[9979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:51 dist-test-slave-45dx krb5kdc[9979](info): set up 2 sockets
Aug 10 06:29:51 dist-test-slave-45dx krb5kdc[9979](info): commencing operation
krb5kdc: starting...
W20260810 06:29:52.754423 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:52 dist-test-slave-45dx krb5kdc[9979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:52.883159 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:41157
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:41157,127.4.73.61:45587,127.4.73.60:34277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:52.991919 10000 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:29:52.992198 10000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:52.992221 10000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:52.992332 10000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:52.996183 10000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:52.996243 10000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:52.996263 10000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:52.996279 10000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:52.996372 10000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:53.001859 10000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:41157,127.4.73.61:45587,127.4.73.60:34277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:41157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:53.002918 10000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:53.003986 10000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:53.012359 10008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:53.012359 10010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:53.012388 10005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:53.012523 10000 server_base.cc:1034] running on GCE node
I20260810 06:29:53.013254 10000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:53.014288 10000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:53.015450 10000 hybrid_clock.cc:648] HybridClock initialized: now 1786343393015431 us; error 30 us; skew 500 ppm
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:53.018157 10000 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:53.018997 10000 webserver.cc:460] Webserver started at http://127.4.73.62:41437/ using document root <none> and password file <none>
I20260810 06:29:53.019541 10000 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:53.019634 10000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:53.019874 10000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:53.022027 10000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "bb6ff55a2b944b8caf849d979abeb393"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "17576bc7d18cc8ead7de4c1a6fa30246"
server_key_iv: "f9bfd66cf6272d0afcb9458d5cdb6357"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.022617 10000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "bb6ff55a2b944b8caf849d979abeb393"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "17576bc7d18cc8ead7de4c1a6fa30246"
server_key_iv: "f9bfd66cf6272d0afcb9458d5cdb6357"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.026422 10000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 06:29:53.028831 10024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:53.030026 10000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:53.030172 10000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "bb6ff55a2b944b8caf849d979abeb393"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "17576bc7d18cc8ead7de4c1a6fa30246"
server_key_iv: "f9bfd66cf6272d0afcb9458d5cdb6357"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.030288 10000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:53.042050 10000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:53.045262 10000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:53.045509 10000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:53.052969 10000 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:41157
I20260810 06:29:53.053260 10111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:41157 every 8 connection(s)
I20260810 06:29:53.053987 10000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:53.056763 10113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:53.059078 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10000
I20260810 06:29:53.059249 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:53.059536 4388 external_mini_cluster.cc:1428] Setting key 3d7d41edfba6e2c0fdf466304589286c
I20260810 06:29:53.060694 10113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.73.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.61@KRBTEST.COM" created.
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:53.078168 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.063300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.4.73.62:41157)
0810 06:29:53.063725 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.063744 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.064019 (+ 275us) server_negotiation.cc:407] Connection header received
0810 06:29:53.065160 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.065164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.065335 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.065533 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.066401 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.067288 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.068324 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.068751 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.068929 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.072788 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.072794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.072805 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.072827 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.074904 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.075459 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.075465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.075467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.075588 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.075983 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.075985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.075986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.076443 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.076553 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.077112 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.077342 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
I20260810 06:29:53.078176 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.063301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:41157 (local address 127.0.0.1:59570)
0810 06:29:53.063867 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.063902 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.064846 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.065541 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.065547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.066004 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.066245 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.066258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.067434 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.067437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.068167 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.068176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.068922 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.068926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.069046 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.069685 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.069701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.072630 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.075036 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.075045 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.075054 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.075291 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.075690 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.075696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.075701 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.075875 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.076568 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.076578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.076797 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.077017 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.077428 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":34,"spinlock_wait_cycles":8448,"thread_start_us":159,"threads_started":1}
I20260810 06:29:53.080803 10113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:53.081594 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.081335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42252)
0810 06:29:53.081467 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.081536 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 06:29:53.081918 10029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
I20260810 06:29:53.082775 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.082559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42264)
0810 06:29:53.082653 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.082695 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 06:29:53.082973 10113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:45587: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab.
Entry for principal kudu/127.4.73.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab.
I20260810 06:29:53.091274 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:45587
--webserver_interface=127.4.73.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab
--principal=kudu/127.4.73.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:41157,127.4.73.61:45587,127.4.73.60:34277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:29:53.132263 10113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } attempt: 1
I20260810 06:29:53.133338 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.132941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42280)
0810 06:29:53.133155 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.133237 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 06:29:53.134156 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.133920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42286)
0810 06:29:53.134035 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.134093 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 06:29:53.134307 10113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:45587: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
I20260810 06:29:53.202472 10113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } attempt: 2
I20260810 06:29:53.203415 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.203062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42292)
0810 06:29:53.203240 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.203317 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 06:29:53.204241 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.203964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42302)
0810 06:29:53.204090 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.204153 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 06:29:53.204425 10113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:45587: Network error: Client connection negotiation failed: client connection to 127.4.73.61:45587: connect: Connection refused (error 111)
W20260810 06:29:53.207365 10131 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:29:53.207744 10131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:53.207809 10131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:53.207892 10131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:53.212276 10131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:53.212452 10131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:53.212491 10131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:53.212529 10131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:53.212560 10131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:53.218477 10131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:41157,127.4.73.61:45587,127.4.73.60:34277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab
--principal=kudu/127.4.73.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:45587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.73.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:53.219712 10131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:53.220876 10131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:53.229667 10139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:53.229791 10131 server_base.cc:1034] running on GCE node
W20260810 06:29:53.229651 10138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:53.229655 10144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:53.230654 10131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:53.231849 10131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:53.233057 10131 hybrid_clock.cc:648] HybridClock initialized: now 1786343393233025 us; error 44 us; skew 500 ppm
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:53.236133 10131 init.cc:377] Logged in from keytab as kudu/127.4.73.61@KRBTEST.COM (short username kudu)
I20260810 06:29:53.237162 10131 webserver.cc:460] Webserver started at http://127.4.73.61:46545/ using document root <none> and password file <none>
I20260810 06:29:53.237787 10131 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:53.237900 10131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:53.238153 10131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:53.240269 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data/instance:
uuid: "e1126db7aeba4edfa14ab5e7efb5366b"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "7d5a63a7e6fe410b7c1e0e9e5ee90368"
server_key_iv: "5be3f7736f69603ed6a92ae24dc86f2f"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.240957 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/wal/instance:
uuid: "e1126db7aeba4edfa14ab5e7efb5366b"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "7d5a63a7e6fe410b7c1e0e9e5ee90368"
server_key_iv: "5be3f7736f69603ed6a92ae24dc86f2f"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.244874 10131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 06:29:53.247381 10157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:53.248780 10131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 06:29:53.248927 10131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/wal
uuid: "e1126db7aeba4edfa14ab5e7efb5366b"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "7d5a63a7e6fe410b7c1e0e9e5ee90368"
server_key_iv: "5be3f7736f69603ed6a92ae24dc86f2f"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.249042 10131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:53.261813 10131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:53.265034 10131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:53.265254 10131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:53.272799 10131 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.61:45587
I20260810 06:29:53.273134 10252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.61:45587 every 8 connection(s)
I20260810 06:29:53.273849 10131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
I20260810 06:29:53.276748 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:53.278335 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10131
I20260810 06:29:53.278476 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-1/wal/instance
I20260810 06:29:53.278790 4388 external_mini_cluster.cc:1428] Setting key 5770498dccd46b21563424b474c32942
I20260810 06:29:53.281731 10254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.73.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.60@KRBTEST.COM" created.
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:53.295615 10113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } attempt: 3
I20260810 06:29:53.299060 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.284277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:41157 (local address 127.0.0.1:59586)
0810 06:29:53.284859 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.284904 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.285782 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.286358 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.286364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.286725 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.287224 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.287236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.288096 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.288100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.289124 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.289136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.289898 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.289911 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.290195 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.291048 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.291068 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.293246 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.295603 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.295610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.295629 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.295990 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.296354 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.296357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.296359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.296546 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.297077 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.297083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.297445 (+ 362us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.297633 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.298029 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":167,"threads_started":1}
I20260810 06:29:53.300529 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.283726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.4.73.62:41157)
0810 06:29:53.283871 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.283875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.285027 (+ 1152us) server_negotiation.cc:407] Connection header received
0810 06:29:53.286003 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.286007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.286070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.286179 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.287381 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.287941 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.289269 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.289653 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.289774 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.293508 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.293514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.293518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.293580 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.295444 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.296111 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.296115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.296117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.296218 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.296668 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.296671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.296673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.296938 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.299482 (+ 2544us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.300181 (+ 699us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.300367 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:29:53.301980 10254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
I20260810 06:29:53.310541 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.296101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42306)
0810 06:29:53.296243 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.296260 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.296355 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.297445 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.297451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.297654 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.297818 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.297828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.299185 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.299190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.300123 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.300138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.300957 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.300961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.301076 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.301689 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.301710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.305694 (+ 3984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.308266 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.308271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.308275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.308621 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.309068 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.309071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.309072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.309191 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.309792 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.309795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.309941 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.310037 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.310326 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 06:29:53.310604 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.296496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42306 (local address 127.4.73.61:45587)
0810 06:29:53.296933 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.296946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.296974 (+ 28us) server_negotiation.cc:407] Connection header received
0810 06:29:53.297025 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.297032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.297275 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.297430 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.297958 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.299033 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.300260 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.300811 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.300957 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.305840 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.305868 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.305887 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.305914 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.308140 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.308733 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.308738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.308739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.308820 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.309302 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.309305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.309306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.309681 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.309794 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.310209 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.310366 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":181,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab.
Entry for principal kudu/127.4.73.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab.
I20260810 06:29:53.311650 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.302529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42322)
0810 06:29:53.302702 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.302716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.302787 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.303163 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.303166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.303335 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.303412 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.303417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.304183 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.304189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.304776 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.304784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.305284 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.305287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.305374 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.305934 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.305949 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.307357 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.309915 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.309917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.309918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.310163 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.310481 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.310484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.310486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.310576 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.311046 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.311048 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.311154 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.311277 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.311477 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 06:29:53.311890 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.302637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.4.73.61:45587)
0810 06:29:53.302981 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.302988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.302998 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:29:53.303024 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.303025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.303057 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.303154 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.303520 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.304076 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.304938 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.305175 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.305286 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.307516 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.307526 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.307529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.307550 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.309821 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.310266 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.310270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.310275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.310370 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.310679 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.310682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.310684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.310893 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.311025 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.311400 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.311534 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260810 06:29:53.312500 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:34277
--webserver_interface=127.4.73.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab
--principal=kudu/127.4.73.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:41157,127.4.73.61:45587,127.4.73.60:34277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:29:53.312997 10254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:53.313127 10113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:53.313767 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.313490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49510)
0810 06:29:53.313618 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.313699 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 06:29:53.313827 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.313513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49522)
0810 06:29:53.313699 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.313743 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260810 06:29:53.314198 10162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
I20260810 06:29:53.314644 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.314320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49532)
0810 06:29:53.314514 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.314571 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260810 06:29:53.314774 10113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:34277: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
I20260810 06:29:53.315075 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.314857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49546)
0810 06:29:53.314991 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.315028 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 06:29:53.315250 10254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:34277: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
I20260810 06:29:53.345963 10113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } attempt: 1
I20260810 06:29:53.346872 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.346530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49554)
0810 06:29:53.346708 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.346779 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 06:29:53.347586 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.347365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49558)
0810 06:29:53.347481 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.347531 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 06:29:53.347762 10113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:34277: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
I20260810 06:29:53.364519 10254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } attempt: 1
I20260810 06:29:53.365299 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.365042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49570)
0810 06:29:53.365177 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.365233 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 06:29:53.366074 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.365873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49580)
0810 06:29:53.365987 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.366028 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 06:29:53.366238 10254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:34277: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
I20260810 06:29:53.422958 10113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } attempt: 2
I20260810 06:29:53.423928 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.423526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49582)
0810 06:29:53.423699 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.423768 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 06:29:53.424665 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.424447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49584)
0810 06:29:53.424555 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.424605 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 06:29:53.424805 10113 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:34277: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
W20260810 06:29:53.426061 10265 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:29:53.426296 10265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:53.426345 10265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:53.426404 10265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:53.430032 10265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:53.430078 10265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:53.430095 10265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:53.430147 10265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:53.430197 10265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:53.434398 10254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } attempt: 2
I20260810 06:29:53.435190 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.434893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49590)
0810 06:29:53.435057 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.435125 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 06:29:53.435954 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.435704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49602)
0810 06:29:53.435818 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.435887 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 06:29:53.436146 10254 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:34277: Network error: Client connection negotiation failed: client connection to 127.4.73.60:34277: connect: Connection refused (error 111)
I20260810 06:29:53.436484 10265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:41157,127.4.73.61:45587,127.4.73.60:34277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab
--principal=kudu/127.4.73.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:34277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.73.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:53.437456 10265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:53.438442 10265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:53.446964 10276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:53.446980 10282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:53.447121 10265 server_base.cc:1034] running on GCE node
W20260810 06:29:53.446972 10277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:53.447808 10265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:53.448941 10265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:53.450136 10265 hybrid_clock.cc:648] HybridClock initialized: now 1786343393450110 us; error 36 us; skew 500 ppm
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:53.453192 10265 init.cc:377] Logged in from keytab as kudu/127.4.73.60@KRBTEST.COM (short username kudu)
I20260810 06:29:53.454241 10265 webserver.cc:460] Webserver started at http://127.4.73.60:44285/ using document root <none> and password file <none>
I20260810 06:29:53.454815 10265 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:53.454887 10265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:53.455066 10265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:53.457269 10265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data/instance:
uuid: "996361d2e33944309118dd37fb79d210"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "86d7b933aa6ee218d37d4a088c23a25e"
server_key_iv: "c6d261932d3c2257cf801f30fa1a5497"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.457809 10265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/wal/instance:
uuid: "996361d2e33944309118dd37fb79d210"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "86d7b933aa6ee218d37d4a088c23a25e"
server_key_iv: "c6d261932d3c2257cf801f30fa1a5497"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.461601 10265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 06:29:53.463925 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:53.465168 10265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:53.465298 10265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/wal
uuid: "996361d2e33944309118dd37fb79d210"
format_stamp: "Formatted at 2026-08-10 06:29:53 on dist-test-slave-45dx"
server_key: "86d7b933aa6ee218d37d4a088c23a25e"
server_key_iv: "c6d261932d3c2257cf801f30fa1a5497"
server_key_version: "encryptionkey@0"
I20260810 06:29:53.465415 10265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:53.476733 10265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:53.479447 10265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:53.479650 10265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:53.486639 10265 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.60:34277
I20260810 06:29:53.486908 10383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.60:34277 every 8 connection(s)
I20260810 06:29:53.487587 10265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
I20260810 06:29:53.490319 10384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:53.490391 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10265
I20260810 06:29:53.490496 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786343096561320-4388-0/minicluster-data/master-2/wal/instance
I20260810 06:29:53.490808 4388 external_mini_cluster.cc:1428] Setting key acfd93198044c832f9576022a6098874
I20260810 06:29:53.495177 10384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:53.505231 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.492100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.4.73.60:34277)
0810 06:29:53.492505 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.492528 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.492562 (+ 34us) server_negotiation.cc:407] Connection header received
0810 06:29:53.493161 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.493176 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.493468 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.493721 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.494684 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.495719 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.496845 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.497600 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.497757 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.500491 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.500496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.500505 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.500532 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.502306 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.502799 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.502808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.502817 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.502974 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.503269 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.503272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.503273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.503614 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.503751 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.504126 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.504453 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":151,"threads_started":1}
I20260810 06:29:53.506708 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.496906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:41157 (local address 127.0.0.1:59596)
0810 06:29:53.497318 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.497366 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.497484 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.497794 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.497800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.498132 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.498251 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.498257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.499015 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.499018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.499979 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.499990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.500501 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.500505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.500655 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.501250 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.501270 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.502904 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.504983 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.504991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.504995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.505278 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.505704 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.505706 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.505708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.505799 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.506159 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.506165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.506276 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.506411 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.506584 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
I20260810 06:29:53.506794 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.497004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.4.73.62:41157)
0810 06:29:53.497148 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.497152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.497450 (+ 298us) server_negotiation.cc:407] Connection header received
0810 06:29:53.497585 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.497589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.497641 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.497738 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.498379 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.498887 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.500097 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.500365 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.500483 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.503060 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.503064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.503066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.503094 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.504876 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.505531 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.505534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.505535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.505611 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.505887 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.505889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.505890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.506059 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.506156 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.506536 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.506654 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 06:29:53.507846 10384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:53.516089 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.508352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:45587 (local address 127.0.0.1:42334)
0810 06:29:53.508490 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.508500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.508582 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.508841 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.508844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.509004 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.509102 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.509107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.509852 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.509855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.510354 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.510361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.510847 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.510850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.510940 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.511347 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.511357 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.512553 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.514558 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.514563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.514567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.514797 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.515069 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.515072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.515074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.515173 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.515602 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.515605 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.515696 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.515799 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.515926 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 06:29:53.516161 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.508478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.4.73.61:45587)
0810 06:29:53.508639 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.508645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.508658 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:53.508698 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.508700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.508744 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.508828 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.509224 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.509739 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.510480 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.510733 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.510845 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.512683 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.512686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.512687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.512711 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.514450 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.514893 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.514895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.514896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.514978 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.515269 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.515276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.515279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.515495 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.515609 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.515866 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.516023 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 06:29:53.516901 10384 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:53.517248 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.507821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.4.73.62:41157)
0810 06:29:53.507920 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.507924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.507938 (+ 14us) server_negotiation.cc:407] Connection header received
0810 06:29:53.508050 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.508054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.508092 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.508182 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.508857 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.509334 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.510090 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.510304 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.510415 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.513409 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.513412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.513414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.513439 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.515104 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.515602 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.515605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.515607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.515690 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.516019 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.516022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.516024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.516241 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.516540 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.516952 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.517074 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
I20260810 06:29:53.524008 10113 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } attempt: 3
I20260810 06:29:53.525782 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.517402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49614)
0810 06:29:53.517486 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.517499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.517568 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.517807 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.517810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.517965 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.518021 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.518025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.519346 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.519349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.520124 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.520135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.520725 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.520727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.520786 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.521234 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.521248 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.522484 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.524189 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.524195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.524198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.524491 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.524775 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.524781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.524786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.524897 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.525320 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.525322 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.525407 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.525508 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.525630 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 06:29:53.525807 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.517535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.4.73.60:34277)
0810 06:29:53.517635 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.517640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.517650 (+ 10us) server_negotiation.cc:407] Connection header received
0810 06:29:53.517676 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.517679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.517717 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.517796 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.518127 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.518907 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.520239 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.520618 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.520723 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.522617 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.522620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.522622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.522643 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.524073 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.524598 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.524604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.524609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.524689 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.525000 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.525003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.525005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.525210 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.525318 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.525552 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.525685 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":35}
I20260810 06:29:53.527294 10254 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } attempt: 3
I20260810 06:29:53.527925 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.518311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.4.73.61:45587)
0810 06:29:53.518418 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.518427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.518439 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:29:53.518533 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.518535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.518565 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.518646 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.519318 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.520069 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.521063 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.521288 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.521398 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.524175 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.524180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.524183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.524210 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.526008 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.526458 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.526462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.526463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.526552 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.526841 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.526844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.526846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.527014 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.527225 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.527669 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.527776 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
I20260810 06:29:53.531581 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:53.531906 10384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: Bootstrap starting.
Aug 10 06:29:53 dist-test-slave-45dx krb5kdc[9979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343393, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
I20260810 06:29:53.533766 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.524396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49618)
0810 06:29:53.524577 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.524590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.524655 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.525175 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.525181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.525369 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.525495 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.525504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.526547 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.526550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.527087 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.527096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.527710 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.527714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.527805 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.528214 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.528227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.529678 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.532098 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.532102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.532104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.532382 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.532696 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.532699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.532701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.532797 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.533240 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.533243 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.533343 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.533439 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.533616 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 06:29:53.534412 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.524498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.4.73.60:34277)
0810 06:29:53.524934 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.524941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.524974 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:29:53.525001 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.525003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.525039 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.525128 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.525634 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.526417 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.527209 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.527565 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.527706 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.529819 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.529834 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.529837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.529859 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.531970 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.532505 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.532508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.532510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.532591 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.532918 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.532921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.532923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.533114 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.534055 (+ 941us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.534130 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.534244 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":113,"threads_started":1}
I20260810 06:29:53.533195 10070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bb6ff55a2b944b8caf849d979abeb393"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59604
I20260810 06:29:53.535540 10384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:53.536466 10384 log.cc:824] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:53.536814 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.527855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.4.73.60:34277)
0810 06:29:53.527972 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.527977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.527990 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:53.528371 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.528375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.528417 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.528485 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.528999 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.529609 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.530645 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.530947 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.531379 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.533196 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.533200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.533203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.533233 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.535040 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.535593 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.535596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.535598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.535675 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.535985 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.535990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.535992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.536197 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.536294 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.536585 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.536687 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 06:29:53.536815 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.527647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:34277 (local address 127.0.0.1:49624)
0810 06:29:53.527877 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:53.527890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:53.528198 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:53.528584 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:53.528607 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:53.528775 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:53.528870 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.528877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.529736 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.529740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.530493 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:53.530504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.531073 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.531076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:53.531166 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.531585 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:53.531603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:53.533043 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:53.535175 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.535179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.535180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.535440 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.535788 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:53.535792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:53.535794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:53.535882 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:53.536327 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:53.536333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:53.536425 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 06:29:53.536534 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 06:29:53.536665 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260810 06:29:53.538152 10113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393: Bootstrap starting.
I20260810 06:29:53.538633 10384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: No bootstrap required, opened a new log
I20260810 06:29:53.539044 10254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: Bootstrap starting.
I20260810 06:29:53.540665 10113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:53.541038 10254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:53.541489 10113 log.cc:824] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:53.541893 10254 log.cc:824] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:53.542763 10384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.543025 10384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996361d2e33944309118dd37fb79d210, State: Initialized, Role: FOLLOWER
I20260810 06:29:53.543386 10384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.543607 10113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393: No bootstrap required, opened a new log
I20260810 06:29:53.543805 10254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: No bootstrap required, opened a new log
I20260810 06:29:53.543995 10403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.544268 10403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:53.544754 10384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:53.546494 10113 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.546692 10113 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6ff55a2b944b8caf849d979abeb393, State: Initialized, Role: FOLLOWER
I20260810 06:29:53.546977 10113 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.547663 10254 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.547852 10254 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1126db7aeba4edfa14ab5e7efb5366b, State: Initialized, Role: FOLLOWER
I20260810 06:29:53.548295 10113 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:53.548377 10070 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 06:29:53.548511 10070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:53.549712 10070 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.550586 10070 leader_election.cc:288] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [CANDIDATE]: Term 1 election: Requested vote from peers e1126db7aeba4edfa14ab5e7efb5366b (127.4.73.61:45587), 996361d2e33944309118dd37fb79d210 (127.4.73.60:34277)
I20260810 06:29:53.551268 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6ff55a2b944b8caf849d979abeb393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "996361d2e33944309118dd37fb79d210"
I20260810 06:29:53.551482 10334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:53.548770 10265 master_runner.cc:186] Error getting master registration for 127.4.73.62:41157: OK, instance_id { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" instance_seqno: 1786343393051019 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:29:53.552125 10265 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:29:53.552346 10254 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.552737 10334 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6ff55a2b944b8caf849d979abeb393 in term 1.
W20260810 06:29:53.553151 10439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:53.553220 10439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:53.553635 10254 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:53.553658 10028 leader_election.cc:302] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 996361d2e33944309118dd37fb79d210, bb6ff55a2b944b8caf849d979abeb393; no voters:
I20260810 06:29:53.553939 10203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb6ff55a2b944b8caf849d979abeb393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1126db7aeba4edfa14ab5e7efb5366b"
I20260810 06:29:53.554157 10203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:53.554167 10452 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.554287 10452 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:53.554375 10452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:53.554471 10452 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [term 1 LEADER]: Becoming Leader. State: Replica: bb6ff55a2b944b8caf849d979abeb393, State: Running, Role: LEADER
I20260810 06:29:53.554788 10452 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } }
I20260810 06:29:53.555505 10203 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb6ff55a2b944b8caf849d979abeb393 in term 1.
W20260810 06:29:53.556778 10421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:53.556922 10421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:53.555540 10000 master_runner.cc:186] Error getting master registration for 127.4.73.61:45587: OK, instance_id { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" instance_seqno: 1786343393270729 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:29:53.558718 10000 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 06:29:53.559092 10470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:53.559995 10470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:53.563046 10423 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6ff55a2b944b8caf849d979abeb393. Latest consensus state: current_term: 1 leader_uuid: "bb6ff55a2b944b8caf849d979abeb393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.563228 10423 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:53.564474 10451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.564666 10451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:53.570510 10478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:53.571734 10478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:53.576035 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.556324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.4.73.61:45587)
0810 06:29:53.556451 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.556458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.559403 (+ 2945us) server_negotiation.cc:407] Connection header received
0810 06:29:53.559600 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.559603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.559651 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.559706 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.560937 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.561701 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.563303 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.563672 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.563745 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.565701 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.565707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.565709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.565735 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.567522 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.570070 (+ 2548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.570074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.570076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.570169 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.575073 (+ 4904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.575077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.575078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.575296 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.575419 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.575764 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.575878 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:29:53.577107 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.559060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.4.73.62:41157)
0810 06:29:53.559171 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.559177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.563447 (+ 4270us) server_negotiation.cc:407] Connection header received
0810 06:29:53.563498 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.563500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.563557 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.563647 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.567731 (+ 4084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.568404 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.569426 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.569722 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.569800 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.571501 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.571505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.571508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.571533 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.573224 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.573746 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.573749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.573751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.573839 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.574380 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.574383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.574385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.574603 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.574707 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.576804 (+ 2097us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.576915 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 06:29:53.580740 10334 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [term 1 FOLLOWER]: Refusing update from remote peer bb6ff55a2b944b8caf849d979abeb393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:53.581439 10452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:53.582894 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:53.557209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.4.73.60:34277)
0810 06:29:53.557310 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.557314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.568543 (+ 11229us) server_negotiation.cc:407] Connection header received
0810 06:29:53.568738 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.568742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.568787 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.568845 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.570694 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.571456 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.576687 (+ 5231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:53.577050 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:53.577121 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:53.579165 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:53.579169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:53.579171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:53.579200 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:53.580939 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.581462 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.581468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.581473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.581562 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:53.581929 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:53.581932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:53.581934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:53.582169 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:53.582277 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:53.582639 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 06:29:53.582760 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 06:29:53.585168 10403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6ff55a2b944b8caf849d979abeb393. Latest consensus state: current_term: 1 leader_uuid: "bb6ff55a2b944b8caf849d979abeb393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.585292 10403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:53.586872 10203 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [term 1 FOLLOWER]: Refusing update from remote peer bb6ff55a2b944b8caf849d979abeb393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:53.587472 10452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:53.588807 10452 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb6ff55a2b944b8caf849d979abeb393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.588918 10452 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:53.590035 10403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb6ff55a2b944b8caf849d979abeb393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.590157 10403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:53.590561 10479 mvcc.cc:205] Tried to move back new op lower bound from 7316862540098502656 to 7316862540023967744. Current Snapshot: MvccSnapshot[applied={T|T < 7316862540098502656}]
I20260810 06:29:53.590924 10480 mvcc.cc:205] Tried to move back new op lower bound from 7316862540098502656 to 7316862540023967744. Current Snapshot: MvccSnapshot[applied={T|T < 7316862540098502656}]
I20260810 06:29:53.591598 10451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6ff55a2b944b8caf849d979abeb393. Latest consensus state: current_term: 1 leader_uuid: "bb6ff55a2b944b8caf849d979abeb393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.591766 10451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:53.600543 10478 catalog_manager.cc:1347] Generated new cluster ID: 5bff9f307cda43819229edcdd7d8099c
I20260810 06:29:53.600611 10478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:53.620739 10423 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb6ff55a2b944b8caf849d979abeb393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.620918 10423 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393 [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:53.622159 10451 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb6ff55a2b944b8caf849d979abeb393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6ff55a2b944b8caf849d979abeb393" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 41157 } } peers { permanent_uuid: "e1126db7aeba4edfa14ab5e7efb5366b" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 45587 } } peers { permanent_uuid: "996361d2e33944309118dd37fb79d210" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 34277 } } }
I20260810 06:29:53.622298 10451 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:53.622746 10485 mvcc.cc:205] Tried to move back new op lower bound from 7316862540098502656 to 7316862540023967744. Current Snapshot: MvccSnapshot[applied={T|T < 7316862540098502656}]
I20260810 06:29:53.650408 10478 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:53.652632 10478 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:53.675264 10478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb6ff55a2b944b8caf849d979abeb393: Generated new TSK 0
I20260810 06:29:53.676544 10478 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:29:53.711666 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.708421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42352 (local address 127.4.73.61:45587)
0810 06:29:53.708556 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.708561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.710536 (+ 1975us) server_negotiation.cc:407] Connection header received
0810 06:29:53.710737 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.710741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.710793 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.710864 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.711535 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42352: BlockingRecv error: recv got EOF from 127.0.0.1:42352 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 06:29:53.713913 10119 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.709248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59616 (local address 127.4.73.62:41157)
0810 06:29:53.709360 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.709365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.712378 (+ 3013us) server_negotiation.cc:407] Connection header received
0810 06:29:53.712886 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.712890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.712942 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.712996 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.713776 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59616: BlockingRecv error: recv got EOF from 127.0.0.1:59616 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 06:29:53.714830 10388 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.709879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49642 (local address 127.4.73.60:34277)
0810 06:29:53.709988 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.709992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.712600 (+ 2608us) server_negotiation.cc:407] Connection header received
0810 06:29:53.713142 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.713146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.713199 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.713253 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.714700 (+ 1447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49642: BlockingRecv error: recv got EOF from 127.0.0.1:49642 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 06:29:53.719725 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.718158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.4.73.61:45587)
0810 06:29:53.718264 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.718268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.718773 (+ 505us) server_negotiation.cc:407] Connection header received
0810 06:29:53.719012 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.719016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.719088 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.719141 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.719628 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42364: BlockingRecv error: recv got EOF from 127.0.0.1:42364 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 06:29:53.724768 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.723182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.4.73.61:45587)
0810 06:29:53.723284 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.723287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.723839 (+ 552us) server_negotiation.cc:407] Connection header received
0810 06:29:53.724039 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.724042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.724091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.724145 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.724676 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42370: BlockingRecv error: recv got EOF from 127.0.0.1:42370 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 06:29:53.730363 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.728870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.4.73.61:45587)
0810 06:29:53.728976 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.728980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.729478 (+ 498us) server_negotiation.cc:407] Connection header received
0810 06:29:53.729691 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.729694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.729744 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.729792 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.730277 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42380: BlockingRecv error: recv got EOF from 127.0.0.1:42380 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 06:29:53.741436 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.736352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.4.73.61:45587)
0810 06:29:53.736481 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.736485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.740598 (+ 4113us) server_negotiation.cc:407] Connection header received
0810 06:29:53.740770 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.740773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.740816 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.740875 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.741342 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42390: BlockingRecv error: recv got EOF from 127.0.0.1:42390 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 06:29:53.753577 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.751742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.4.73.61:45587)
0810 06:29:53.751860 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.751864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.752503 (+ 639us) server_negotiation.cc:407] Connection header received
0810 06:29:53.752701 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.752704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.752749 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.752797 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.753474 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42402: BlockingRecv error: recv got EOF from 127.0.0.1:42402 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 06:29:53.772962 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.771707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.4.73.61:45587)
0810 06:29:53.771820 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.771834 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.772075 (+ 241us) server_negotiation.cc:407] Connection header received
0810 06:29:53.772235 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.772238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.772289 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.772407 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.772847 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42412: BlockingRecv error: recv got EOF from 127.0.0.1:42412 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 06:29:53.808132 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.807063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.4.73.61:45587)
0810 06:29:53.807231 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.807235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.807286 (+ 51us) server_negotiation.cc:407] Connection header received
0810 06:29:53.807413 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.807415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.807462 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.807551 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.808010 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42420: BlockingRecv error: recv got EOF from 127.0.0.1:42420 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 06:29:53.875735 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:53.874388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.4.73.61:45587)
0810 06:29:53.874569 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:53.874574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:53.874733 (+ 159us) server_negotiation.cc:407] Connection header received
0810 06:29:53.874943 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:53.874946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:53.874998 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:53.875091 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:53.875586 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42422: BlockingRecv error: recv got EOF from 127.0.0.1:42422 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 06:29:54.007359 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:54.006054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.4.73.61:45587)
0810 06:29:54.006212 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:54.006216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:54.006412 (+ 196us) server_negotiation.cc:407] Connection header received
0810 06:29:54.006537 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:54.006539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:54.006586 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:54.006670 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:54.007218 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42428: BlockingRecv error: recv got EOF from 127.0.0.1:42428 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 06:29:54.267164 10262 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:54.265843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.4.73.61:45587)
0810 06:29:54.266010 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:54.266016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:54.266098 (+ 82us) server_negotiation.cc:407] Connection header received
0810 06:29:54.266298 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:54.266302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:54.266361 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:54.266462 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:54.267013 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42434: BlockingRecv error: recv got EOF from 127.0.0.1:42434 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 06:29:54.558300 10421 catalog_manager.cc:1259] Loaded cluster ID: 5bff9f307cda43819229edcdd7d8099c
I20260810 06:29:54.558384 10421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: loading cluster ID for follower catalog manager: success
I20260810 06:29:54.560019 10421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: acquiring CA information for follower catalog manager: success
I20260810 06:29:54.561079 10470 catalog_manager.cc:1259] Loaded cluster ID: 5bff9f307cda43819229edcdd7d8099c
I20260810 06:29:54.561142 10470 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: loading cluster ID for follower catalog manager: success
I20260810 06:29:54.561303 10421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 996361d2e33944309118dd37fb79d210: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:29:54.562711 10470 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: acquiring CA information for follower catalog manager: success
I20260810 06:29:54.563798 10470 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1126db7aeba4edfa14ab5e7efb5366b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:29:54.785077 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:54.781299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.4.73.61:45587)
0810 06:29:54.781509 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:54.781512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:54.781582 (+ 70us) server_negotiation.cc:407] Connection header received
0810 06:29:54.781701 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:54.781704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:54.781843 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:54.781953 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:54.782192 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.782715 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.783537 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.783752 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.783831 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:54.784115 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:54.784685 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:54.784778 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:54.784839 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 06:29:54.784884 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260810 06:29:54.791419 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:54.788031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.4.73.60:34277)
0810 06:29:54.788300 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:54.788345 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:54.788369 (+ 24us) server_negotiation.cc:407] Connection header received
0810 06:29:54.788476 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:54.788479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:54.788579 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:54.788672 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:54.788860 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.789350 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.790146 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.790398 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.790483 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:54.790764 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:54.791106 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:54.791198 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:54.791265 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 06:29:54.791308 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":68,"threads_started":1}
I20260810 06:29:54.798359 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:54.794442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.4.73.61:45587)
0810 06:29:54.794569 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:54.794573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:54.794821 (+ 248us) server_negotiation.cc:407] Connection header received
0810 06:29:54.794923 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:54.794926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:54.794973 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:54.795037 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:54.795362 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.796043 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.797126 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.797460 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.797549 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:54.797858 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:54.797935 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:54.798017 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:54.798145 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 06:29:54.798199 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 06:29:54.799049 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:54.794690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.4.73.60:34277)
0810 06:29:54.794803 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:54.794808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:54.795207 (+ 399us) server_negotiation.cc:407] Connection header received
0810 06:29:54.795435 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:54.795438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:54.795495 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:54.795565 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:54.795842 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.796570 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.797526 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.797833 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.797929 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:54.798563 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:54.798629 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:54.798714 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:54.798883 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 06:29:54.798933 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 06:29:54.799116 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:54.794362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.4.73.62:41157)
0810 06:29:54.794722 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:54.794726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:54.794741 (+ 15us) server_negotiation.cc:407] Connection header received
0810 06:29:54.794946 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:54.794950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:54.795096 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:54.795230 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:54.795526 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.796399 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.797469 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:54.797728 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:54.797847 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:54.798367 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:54.798639 (+ 272us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:54.798721 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:54.798872 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 06:29:54.798929 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":74,"threads_started":1}
I20260810 06:29:54.801997 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10000
I20260810 06:29:54.818476 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10131
I20260810 06:29:54.831295 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10265
2026-08-10T06:29:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3349 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 06:29:54 dist-test-slave-45dx krb5kdc[10612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 06:29:54 dist-test-slave-45dx krb5kdc[10612](info): set up 2 sockets
Aug 10 06:29:54 dist-test-slave-45dx krb5kdc[10612](info): commencing operation
krb5kdc: starting...
W20260810 06:29:56.103262 4388 mini_kdc.cc:121] Time spent starting KDC: real 1.246s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/test-user.keytab.
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T06:29:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T06:29:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.73.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
Entry for principal kudu/127.4.73.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab.
I20260810 06:29:56.231782 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:37585
--webserver_interface=127.4.73.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:37585,127.4.73.61:44369,127.4.73.60:33921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 06:29:56.345630 10634 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:29:56.345932 10634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:56.345957 10634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:56.345994 10634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:56.349951 10634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:56.350003 10634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:56.350023 10634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:56.350039 10634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:56.350054 10634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:56.355235 10634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:37585,127.4.73.61:44369,127.4.73.60:33921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.62.keytab
--principal=kudu/127.4.73.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.62:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.73.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:56.356189 10634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:56.357247 10634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 06:29:56.366216 10634 server_base.cc:1034] running on GCE node
W20260810 06:29:56.366122 10650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:56.366130 10646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:56.366122 10647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:56.367002 10634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:56.368145 10634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:56.369356 10634 hybrid_clock.cc:648] HybridClock initialized: now 1786343396369342 us; error 29 us; skew 500 ppm
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:56.372375 10634 init.cc:377] Logged in from keytab as kudu/127.4.73.62@KRBTEST.COM (short username kudu)
I20260810 06:29:56.373306 10634 webserver.cc:460] Webserver started at http://127.4.73.62:46421/ using document root <none> and password file <none>
I20260810 06:29:56.373874 10634 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:56.373937 10634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:56.374171 10634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:56.376216 10634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data/instance:
uuid: "401c2617c1aa42b7910935b20764052b"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "0ef2c08c08f1061b9ef44b2bd86e1220"
server_key_iv: "1ae7d19edb1a4e878d6bd92993ae824a"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.376883 10634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/wal/instance:
uuid: "401c2617c1aa42b7910935b20764052b"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "0ef2c08c08f1061b9ef44b2bd86e1220"
server_key_iv: "1ae7d19edb1a4e878d6bd92993ae824a"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.380514 10634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 06:29:56.382838 10660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:56.383994 10634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:56.384123 10634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/wal
uuid: "401c2617c1aa42b7910935b20764052b"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "0ef2c08c08f1061b9ef44b2bd86e1220"
server_key_iv: "1ae7d19edb1a4e878d6bd92993ae824a"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.384230 10634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:56.405144 10634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:56.408747 10634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:56.408979 10634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:56.416529 10634 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.62:37585
I20260810 06:29:56.416858 10750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.62:37585 every 8 connection(s)
I20260810 06:29:56.417544 10634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/data/info.pb
I20260810 06:29:56.418151 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10634
I20260810 06:29:56.418286 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-0/wal/instance
I20260810 06:29:56.418593 4388 external_mini_cluster.cc:1428] Setting key 24d8eaa622db2c31b4de6101f244380a
I20260810 06:29:56.420964 10751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:56.426697 10751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.73.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.61@KRBTEST.COM" created.
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:56.446343 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.429757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:37585 (local address 127.0.0.1:47110)
0810 06:29:56.430385 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.430446 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.431623 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.432952 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.432959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.433606 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.433938 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.433956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.435858 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.435863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.437209 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.437224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.438172 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.438175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.438313 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.438967 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.438984 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.440654 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.443084 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.443092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.443101 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.443334 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.443753 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.443759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.443761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.443964 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.444768 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.444774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.444969 (+ 195us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.445148 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.445564 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":102,"spinlock_wait_cycles":16768,"thread_start_us":201,"threads_started":1}
I20260810 06:29:56.446346 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.429757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.4.73.62:37585)
0810 06:29:56.430246 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.430267 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.430601 (+ 334us) server_negotiation.cc:407] Connection header received
0810 06:29:56.432195 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.432203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.432633 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.432902 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.434129 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.435665 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.437341 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.438003 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.438263 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.440833 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.440842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.440862 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.440898 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.442965 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.443466 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.443474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.443478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.443626 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.444070 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.444074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.444075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.444631 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.444746 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.445226 (+ 480us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.445445 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":318,"spinlock_wait_cycles":2816,"thread_start_us":122,"threads_started":1}
I20260810 06:29:56.449126 10751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:56.450116 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.449681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34516)
0810 06:29:56.449880 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.450006 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260810 06:29:56.450397 10662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
I20260810 06:29:56.451366 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.451085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34532)
0810 06:29:56.451214 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.451262 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab.
Entry for principal kudu/127.4.73.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab.
W20260810 06:29:56.451579 10751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:44369: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
I20260810 06:29:56.452646 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:44369
--webserver_interface=127.4.73.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab
--principal=kudu/127.4.73.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:37585,127.4.73.61:44369,127.4.73.60:33921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:29:56.500847 10751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } attempt: 1
I20260810 06:29:56.501863 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.501534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34534)
0810 06:29:56.501697 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.501778 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 06:29:56.502696 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.502506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34542)
0810 06:29:56.502601 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.502645 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 06:29:56.502849 10751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:44369: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
W20260810 06:29:56.569173 10761 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:29:56.569540 10761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:56.569619 10761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:56.569703 10761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 06:29:56.571024 10751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } attempt: 2
I20260810 06:29:56.571907 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.571598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34552)
0810 06:29:56.571754 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.571821 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 06:29:56.572749 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.572495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34560)
0810 06:29:56.572637 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.572693 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 06:29:56.572875 10751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.61:44369: Network error: Client connection negotiation failed: client connection to 127.4.73.61:44369: connect: Connection refused (error 111)
W20260810 06:29:56.574390 10761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:56.574476 10761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:56.574522 10761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:56.574574 10761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:56.574610 10761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:56.580081 10761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:37585,127.4.73.61:44369,127.4.73.60:33921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.61.keytab
--principal=kudu/127.4.73.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.61:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.73.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:56.581254 10761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:56.582578 10761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:56.591563 10773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:56.591595 10776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:56.591581 10772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:56.591789 10761 server_base.cc:1034] running on GCE node
I20260810 06:29:56.592625 10761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:56.593793 10761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:56.595007 10761 hybrid_clock.cc:648] HybridClock initialized: now 1786343396594992 us; error 34 us; skew 500 ppm
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:56.601989 10761 init.cc:377] Logged in from keytab as kudu/127.4.73.61@KRBTEST.COM (short username kudu)
I20260810 06:29:56.603081 10761 webserver.cc:460] Webserver started at http://127.4.73.61:38991/ using document root <none> and password file <none>
I20260810 06:29:56.603719 10761 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:56.603864 10761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:56.604130 10761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:56.606317 10761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data/instance:
uuid: "c34bf8b6539141b5a64b7a29d417f9fb"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "e8665ca8b77426b34acb72952df77aec"
server_key_iv: "2ea6cd472d4915b333301c5444395da7"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.606916 10761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/wal/instance:
uuid: "c34bf8b6539141b5a64b7a29d417f9fb"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "e8665ca8b77426b34acb72952df77aec"
server_key_iv: "2ea6cd472d4915b333301c5444395da7"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.610754 10761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 06:29:56.613345 10785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:56.614583 10761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 06:29:56.614741 10761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/wal
uuid: "c34bf8b6539141b5a64b7a29d417f9fb"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "e8665ca8b77426b34acb72952df77aec"
server_key_iv: "2ea6cd472d4915b333301c5444395da7"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.614878 10761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:56.631371 10761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:56.634339 10761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:56.634574 10761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:56.641834 10761 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.61:44369
I20260810 06:29:56.642166 10859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.61:44369 every 8 connection(s)
I20260810 06:29:56.642835 10761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/data/info.pb
I20260810 06:29:56.645848 10860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:56.649649 10860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:56.650099 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10761
I20260810 06:29:56.650221 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-1/wal/instance
I20260810 06:29:56.650521 4388 external_mini_cluster.cc:1428] Setting key c24c76829d5e0c9960e158bf07dd50c6
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.73.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.73.60@KRBTEST.COM" created.
I20260810 06:29:56.663776 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.651868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.4.73.62:37585)
0810 06:29:56.652070 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.652077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.652873 (+ 796us) server_negotiation.cc:407] Connection header received
0810 06:29:56.653821 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.653827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.653903 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.654017 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.655164 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.655701 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.656982 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.657227 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.657347 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.659871 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.659875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.659878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.659910 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.661488 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.662008 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.662011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.662012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.662155 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.662545 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.662552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.662557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.662777 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.662917 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.663434 (+ 517us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.663600 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 06:29:56.664014 10751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } attempt: 3
I20260810 06:29:56.665530 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.652184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:37585 (local address 127.0.0.1:47114)
0810 06:29:56.652699 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.652751 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.653608 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.654159 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.654166 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.654515 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.654995 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.655008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.655846 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.655854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.656843 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.656853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.657386 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.657391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.657593 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.658237 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.658254 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.659703 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.661629 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.661634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.661645 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.661917 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.662267 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.662269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.662270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.662447 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.662918 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.662926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.663179 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.663373 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.663706 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":187,"threads_started":1}
I20260810 06:29:56.666996 10860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
I20260810 06:29:56.674944 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.664446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34570)
0810 06:29:56.664614 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.664626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.664695 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.665382 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.665389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.665582 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.665712 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.665722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.667216 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.667219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.668295 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.668361 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.669260 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.669264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.669373 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.669889 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.669902 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.671175 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.673342 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.673345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.673346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.673525 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.673789 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.673791 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.673792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.673874 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.674338 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.674341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.674444 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.674558 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.674689 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 06:29:56.675845 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.664569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.4.73.61:44369)
0810 06:29:56.664953 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.664964 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.664985 (+ 21us) server_negotiation.cc:407] Connection header received
0810 06:29:56.665042 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.665047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.665229 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.665373 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.665869 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.667098 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.668469 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.669130 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.669253 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.671342 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.671352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.671364 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.671388 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.673246 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.673615 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.673621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.673623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.673707 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.673990 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.673994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.673996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.674227 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.674378 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.674725 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.674898 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"spinlock_wait_cycles":14080,"thread_start_us":179,"threads_started":1}
I20260810 06:29:56.677911 10751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:56.678277 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.667527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34586)
0810 06:29:56.667641 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.667656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.667741 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.668967 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.668970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.669135 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.669240 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.669247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.670188 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.670194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.671116 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.671129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.671721 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.671724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.671959 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.672395 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.672411 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.674726 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.676727 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.676732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.676734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.676992 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.677359 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.677361 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.677363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.677457 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.677861 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.677864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.677952 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.678063 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.678178 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":89}
I20260810 06:29:56.678520 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.668512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.4.73.61:44369)
0810 06:29:56.668776 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.668782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.668793 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:29:56.668820 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.668822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.668859 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.668958 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.669362 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.670065 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.671251 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.671626 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.671724 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.674855 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.674862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.674867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.674889 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.676624 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.677182 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.677185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.677186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.677263 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.677548 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.677550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.677551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.677747 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.677863 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.678145 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.678258 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260810 06:29:56.678862 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.678395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59810)
0810 06:29:56.678561 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.678803 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.73.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab.
Entry for principal kudu/127.4.73.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab.
I20260810 06:29:56.679147 10860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:56.679618 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.679370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59816)
0810 06:29:56.679525 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.679569 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260810 06:29:56.679756 10751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:33921: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
I20260810 06:29:56.679972 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.679648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59818)
0810 06:29:56.679838 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.679912 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 06:29:56.680154 10787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
I20260810 06:29:56.680411 4388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPhbgll/build/debug/bin/kudu
/tmp/dist-test-taskPhbgll/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:33921
--webserver_interface=127.4.73.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab
--principal=kudu/127.4.73.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.73.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.73.62:37585,127.4.73.61:44369,127.4.73.60:33921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 06:29:56.681048 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.680846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59830)
0810 06:29:56.680949 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.680993 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 06:29:56.681201 10860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:33921: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
I20260810 06:29:56.710922 10751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } attempt: 1
I20260810 06:29:56.711761 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.711437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59840)
0810 06:29:56.711624 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.711689 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 06:29:56.712535 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.712353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59846)
0810 06:29:56.712438 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.712482 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 06:29:56.712692 10751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:33921: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
I20260810 06:29:56.730436 10860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } attempt: 1
I20260810 06:29:56.731310 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.731016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59854)
0810 06:29:56.731168 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.731231 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 06:29:56.732091 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.731855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59864)
0810 06:29:56.731975 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.732031 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 06:29:56.732223 10860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:33921: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
I20260810 06:29:56.787912 10751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } attempt: 2
I20260810 06:29:56.788763 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.788454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59880)
0810 06:29:56.788616 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.788684 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 06:29:56.789465 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.789250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59894)
0810 06:29:56.789357 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.789408 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 06:29:56.789618 10751 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:33921: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
W20260810 06:29:56.792461 10876 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 06:29:56.792748 10876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 06:29:56.792809 10876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 06:29:56.792891 10876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 06:29:56.796895 10876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 06:29:56.796965 10876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 06:29:56.797001 10876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 06:29:56.797056 10876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 06:29:56.797093 10876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 06:29:56.800362 10860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } attempt: 2
I20260810 06:29:56.800984 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.800745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59908)
0810 06:29:56.800864 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.800923 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 06:29:56.801695 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.801485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59924)
0810 06:29:56.801592 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.801639 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 06:29:56.801829 10860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.73.60:33921: Network error: Client connection negotiation failed: client connection to 127.4.73.60:33921: connect: Connection refused (error 111)
I20260810 06:29:56.802522 10876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.73.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.73.62:37585,127.4.73.61:44369,127.4.73.60:33921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/krb5kdc/kudu_127.4.73.60.keytab
--principal=kudu/127.4.73.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.73.60:33921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.73.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 05:31:42 UTC on ccycloud.quasar-clutqi.root.comops.site
build id 4577008
I20260810 06:29:56.803517 10876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 06:29:56.804554 10876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 06:29:56.812654 10890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:56.812668 10894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 06:29:56.812722 10889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 06:29:56.812997 10876 server_base.cc:1034] running on GCE node
I20260810 06:29:56.813580 10876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 06:29:56.814599 10876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 06:29:56.815784 10876 hybrid_clock.cc:648] HybridClock initialized: now 1786343396815764 us; error 31 us; skew 500 ppm
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 06:29:56.818539 10876 init.cc:377] Logged in from keytab as kudu/127.4.73.60@KRBTEST.COM (short username kudu)
I20260810 06:29:56.819427 10876 webserver.cc:460] Webserver started at http://127.4.73.60:33901/ using document root <none> and password file <none>
I20260810 06:29:56.819962 10876 fs_manager.cc:359] Metadata directory not provided
I20260810 06:29:56.820072 10876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 06:29:56.820346 10876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 06:29:56.822436 10876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data/instance:
uuid: "bcf08969367846c88ebc9237047845b5"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "4243412f3ef996b88df3c924987d8cdd"
server_key_iv: "62866a4522fcc14e945e096054e2503d"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.823019 10876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/wal/instance:
uuid: "bcf08969367846c88ebc9237047845b5"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "4243412f3ef996b88df3c924987d8cdd"
server_key_iv: "62866a4522fcc14e945e096054e2503d"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.826727 10876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 06:29:56.829133 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 06:29:56.830332 10876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 06:29:56.830475 10876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data,/tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/wal
uuid: "bcf08969367846c88ebc9237047845b5"
format_stamp: "Formatted at 2026-08-10 06:29:56 on dist-test-slave-45dx"
server_key: "4243412f3ef996b88df3c924987d8cdd"
server_key_iv: "62866a4522fcc14e945e096054e2503d"
server_key_version: "encryptionkey@0"
I20260810 06:29:56.830586 10876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 06:29:56.842171 10876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 06:29:56.845463 10876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 06:29:56.845659 10876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 06:29:56.852499 10876 rpc_server.cc:307] RPC server started. Bound to: 127.4.73.60:33921
I20260810 06:29:56.852828 10994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.73.60:33921 every 8 connection(s)
I20260810 06:29:56.853430 10876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/data/info.pb
I20260810 06:29:56.856335 10997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 06:29:56.856900 4388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPhbgll/build/debug/bin/kudu as pid 10876
I20260810 06:29:56.857043 4388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPhbgll/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786343096561320-4388-0/minicluster-data/master-2/wal/instance
I20260810 06:29:56.857393 4388 external_mini_cluster.cc:1428] Setting key 68696b0514d3bc92a7d9e30eb257a6f7
I20260810 06:29:56.861492 10997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:56.872689 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.858778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.4.73.60:33921)
0810 06:29:56.859216 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.859231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.859264 (+ 33us) server_negotiation.cc:407] Connection header received
0810 06:29:56.859956 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.859972 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.860279 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.860697 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.861512 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.862586 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.863773 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.864567 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.864748 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.867553 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.867558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.867568 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.867593 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.869508 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.870043 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.870055 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.870060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.870172 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.870500 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.870504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.870506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.870924 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.871137 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.871496 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.871808 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":154,"threads_started":1}
I20260810 06:29:56.873520 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.863300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.4.73.62:37585)
0810 06:29:56.863444 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.863449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.863776 (+ 327us) server_negotiation.cc:407] Connection header received
0810 06:29:56.864041 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.864044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.864092 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.864166 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.865103 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.865639 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.866908 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.867178 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.867264 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.869979 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.869982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.869984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.870010 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.871475 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.872078 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.872082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.872084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.872166 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.872502 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.872506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.872508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.872699 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.872821 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.873295 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.873403 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:29:56.873518 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.863214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.62:37585 (local address 127.0.0.1:47118)
0810 06:29:56.863616 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.863658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.863881 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.864246 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.864252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.864777 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.864961 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.864967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.865773 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.865776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.866762 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.866775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.867353 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.867358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.867494 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.868077 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.868090 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.869830 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.871604 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.871614 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.871619 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.871976 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.872286 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.872289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.872291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.872409 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.872925 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.872932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.873051 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.873186 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.873351 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":139,"threads_started":1}
I20260810 06:29:56.874632 10997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.62@KRBTEST.COM
I20260810 06:29:56.883991 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.875144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.61:44369 (local address 127.0.0.1:34600)
0810 06:29:56.875298 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.875312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.875395 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.875708 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.875713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.875914 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.876026 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.876033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.877090 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.877093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.877647 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.877655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.878271 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.878275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.878377 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.878837 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.878852 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.880352 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.882391 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.882394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.882396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.882646 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.882958 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.882960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.882961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.883037 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.883495 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.883503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.883577 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.883698 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.883820 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 06:29:56.884001 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.875262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.4.73.61:44369)
0810 06:29:56.875394 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.875398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.875411 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:56.875529 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.875533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.875598 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.875689 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.876177 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.876972 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.877761 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.878166 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.878250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.880506 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.880510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.880512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.880539 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.882247 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.882748 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.882751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.882752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.882833 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.883132 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.883137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.883141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.883365 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.883490 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.883758 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.883854 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 06:29:56.884636 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.875245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.4.73.62:37585)
0810 06:29:56.875364 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.875369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.875642 (+ 273us) server_negotiation.cc:407] Connection header received
0810 06:29:56.875757 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.875762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.875835 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.875917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.876681 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.877188 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.878066 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.878299 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.878391 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.881070 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.881074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.881077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.881108 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.882797 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.883285 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.883287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.883288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.883369 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.883654 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.883660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.883664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.883883 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.884094 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.884343 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.884461 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 06:29:56.885129 10997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } has no permanent_uuid. Determining permanent_uuid...
I20260810 06:29:56.888769 10751 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } attempt: 3
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.60@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.73.61@KRBTEST.COM
I20260810 06:29:56.892980 10860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } attempt: 3
I20260810 06:29:56.894100 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.885630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.4.73.60:33921)
0810 06:29:56.885730 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.885735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.885807 (+ 72us) server_negotiation.cc:407] Connection header received
0810 06:29:56.885910 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.885915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.885948 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.886035 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.886369 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.886877 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.887674 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.887906 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.888007 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.890123 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.890127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.890129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.890154 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.891839 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.892268 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.892270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.892271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.892394 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.892914 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.892918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.892919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.893131 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.893229 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.893904 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.894005 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 06:29:56.894752 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.885602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59942)
0810 06:29:56.885728 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.885740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.885842 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.886046 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.886049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.886201 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.886257 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.886262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.886973 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.886980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.887568 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.887576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.888007 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.888011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.888079 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.888546 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.888559 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.889988 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.891936 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.891939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.891941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.892175 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.892497 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.892499 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.892502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.892588 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.893386 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.893389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.893465 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.894510 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.894620 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.62@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
I20260810 06:29:56.896250 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.885990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34608 (local address 127.4.73.61:44369)
0810 06:29:56.886089 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.886093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.886105 (+ 12us) server_negotiation.cc:407] Connection header received
0810 06:29:56.886181 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.886184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.886220 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.886311 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.886921 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.887508 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.888277 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.888544 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.888654 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.891414 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.891418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.891420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.891444 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.894091 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.894618 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.894621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.894624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.894710 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.895011 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.895014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.895016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.895217 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.895416 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.896016 (+ 600us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.896126 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 06:29:56.898077 10997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: Bootstrap starting.
I20260810 06:29:56.898958 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.889138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59944)
0810 06:29:56.889238 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.889255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.889328 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.889906 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.889909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.890104 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.890216 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.890223 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.891817 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.891822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.892688 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.892700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.893292 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.893295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.893382 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.893798 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.893813 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.895186 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.897356 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.897359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.897361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.897609 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.897906 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.897909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.897911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.898006 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.898453 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.898456 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.898555 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.898659 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.898798 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 06:29:56.898990 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.889383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.4.73.60:33921)
0810 06:29:56.889697 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.889705 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.889718 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:56.889753 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.889755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.889786 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.889889 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.890367 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.891688 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.892804 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.893180 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.893277 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.895427 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.895430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.895433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.895455 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.897231 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.897708 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.897711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.897713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.897800 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.898106 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.898109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.898111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.898342 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.898458 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.898736 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.898835 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
Aug 10 06:29:56 dist-test-slave-45dx krb5kdc[10612](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786343396, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.73.61@KRBTEST.COM for kudu/127.4.73.60@KRBTEST.COM
I20260810 06:29:56.900872 10997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:56.901780 10997 log.cc:824] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:56.901991 10751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b: Bootstrap starting.
I20260810 06:29:56.903846 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.893561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.4.73.60:33921)
0810 06:29:56.894275 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.894279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.894292 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:56.894324 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.894327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.894372 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.894446 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.895219 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.896153 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.897532 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.897819 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.897918 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.900541 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.900544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.900547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.900575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.902232 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.902709 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.902713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.902715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.902796 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.903088 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.903092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.903093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.903248 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.903356 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.903599 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.903695 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":104,"threads_started":1}
I20260810 06:29:56.903846 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.893336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.73.60:33921 (local address 127.0.0.1:59952)
0810 06:29:56.893461 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 06:29:56.893476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 06:29:56.893810 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 06:29:56.894465 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 06:29:56.894468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 06:29:56.894638 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 06:29:56.894723 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.894729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.896566 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.896570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.897419 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 06:29:56.897427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.898018 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.898020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 06:29:56.898112 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.899142 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 06:29:56.899159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 06:29:56.900379 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 06:29:56.902359 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.902365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.902370 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.902624 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.902898 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 06:29:56.902901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 06:29:56.902903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 06:29:56.902999 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 06:29:56.903350 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 06:29:56.903353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 06:29:56.903446 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 06:29:56.903549 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 06:29:56.903674 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 06:29:56.904008 10997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: No bootstrap required, opened a new log
I20260810 06:29:56.904395 10751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:56.905165 10751 log.cc:824] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:56.907203 10751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b: No bootstrap required, opened a new log
I20260810 06:29:56.908413 4388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 06:29:56.908372 10997 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.908536 10860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: Bootstrap starting.
I20260810 06:29:56.908600 10997 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08969367846c88ebc9237047845b5, State: Initialized, Role: FOLLOWER
I20260810 06:29:56.908948 10997 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.909734 10704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "401c2617c1aa42b7910935b20764052b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47134
I20260810 06:29:56.910235 10997 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:56.911052 10751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.911310 10860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: Neither blocks nor log segments found. Creating new log.
I20260810 06:29:56.911300 10751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401c2617c1aa42b7910935b20764052b, State: Initialized, Role: FOLLOWER
I20260810 06:29:56.911623 10751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.912149 10860 log.cc:824] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: Log is configured to *not* fsync() on all Append() calls
I20260810 06:29:56.912963 11018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.913039 11010 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.913166 11010 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:56.913518 10751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:56.913583 10704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 06:29:56.913697 10704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:56.914559 10860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: No bootstrap required, opened a new log
I20260810 06:29:56.914959 10704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.915781 10704 leader_election.cc:288] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [CANDIDATE]: Term 1 election: Requested vote from peers c34bf8b6539141b5a64b7a29d417f9fb (127.4.73.61:44369), bcf08969367846c88ebc9237047845b5 (127.4.73.60:33921)
I20260810 06:29:56.917006 10955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "401c2617c1aa42b7910935b20764052b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bcf08969367846c88ebc9237047845b5"
I20260810 06:29:56.914094 10876 master_runner.cc:186] Error getting master registration for 127.4.73.62:37585: OK, instance_id { permanent_uuid: "401c2617c1aa42b7910935b20764052b" instance_seqno: 1786343396414415 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 06:29:56.917585 10876 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:29:56.918344 10860 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.918536 10860 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34bf8b6539141b5a64b7a29d417f9fb, State: Initialized, Role: FOLLOWER
I20260810 06:29:56.918861 10860 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.919317 10955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [term 0 FOLLOWER]: Advancing to term 1
W20260810 06:29:56.919446 11051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:56.919516 11051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:56.920073 10860 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 06:29:56.920641 10955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401c2617c1aa42b7910935b20764052b in term 1.
I20260810 06:29:56.921495 11018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:56.921731 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.921865 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:56.921993 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "401c2617c1aa42b7910935b20764052b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c34bf8b6539141b5a64b7a29d417f9fb"
I20260810 06:29:56.922200 10824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 06:29:56.923357 10824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 401c2617c1aa42b7910935b20764052b in term 1.
I20260810 06:29:56.923900 10662 leader_election.cc:302] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401c2617c1aa42b7910935b20764052b, c34bf8b6539141b5a64b7a29d417f9fb; no voters:
I20260810 06:29:56.928352 10634 master_runner.cc:263] No leader master found from master 401c2617c1aa42b7910935b20764052b
I20260810 06:29:56.928417 10634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:29:56.928547 10761 master_runner.cc:263] No leader master found from master c34bf8b6539141b5a64b7a29d417f9fb
I20260810 06:29:56.928612 10761 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 06:29:56.928560 11018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 06:29:56.928884 11018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [term 1 LEADER]: Becoming Leader. State: Replica: 401c2617c1aa42b7910935b20764052b, State: Running, Role: LEADER
I20260810 06:29:56.929164 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.918978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.4.73.61:44369)
0810 06:29:56.919079 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.919084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.919853 (+ 769us) server_negotiation.cc:407] Connection header received
0810 06:29:56.920035 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.920038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.920086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.920159 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.921180 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.921903 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.923038 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.923411 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.923525 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.925247 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.925251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.925254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.925279 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.927077 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.927562 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.927566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.927568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.927654 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.927966 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.927969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.927971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.928179 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.928317 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.928907 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.929020 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 06:29:56.929447 11018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } }
I20260810 06:29:56.931214 11018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401c2617c1aa42b7910935b20764052b. Latest consensus state: current_term: 1 leader_uuid: "401c2617c1aa42b7910935b20764052b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.931406 11018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: This master's current role is: LEADER
W20260810 06:29:56.931989 11080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:56.932072 11080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 06:29:56.934926 11025 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 06:29:56.935004 11025 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 06:29:56.940259 11085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 06:29:56.941347 11085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 06:29:56.945143 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.935288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.4.73.62:37585)
0810 06:29:56.935416 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.935421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.935849 (+ 428us) server_negotiation.cc:407] Connection header received
0810 06:29:56.935893 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.935896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.935947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.936020 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.936811 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.937600 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.938750 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.939105 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.939207 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.941118 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.941121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.941124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.941148 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.942903 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.943401 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.943404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.943406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.943500 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.943815 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.943818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.943819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.944041 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.944156 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.944717 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.944822 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:29:56.946144 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:56.919609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.4.73.60:33921)
0810 06:29:56.921428 (+ 1819us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:56.921432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:56.921448 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:29:56.921487 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:56.921490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:56.921543 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:56.921632 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:56.932610 (+ 10978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.933316 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.934436 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:56.934760 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:56.934837 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:56.936835 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 06:29:56.936838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 06:29:56.936841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 06:29:56.936867 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 06:29:56.938596 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.939069 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.939073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.939074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.939166 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 06:29:56.939490 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 06:29:56.939495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 06:29:56.939497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 06:29:56.940912 (+ 1415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 06:29:56.941017 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:56.945864 (+ 4847us) server_negotiation.cc:299] Negotiation successful
0810 06:29:56.945987 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1735}
I20260810 06:29:56.949767 10955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [term 1 FOLLOWER]: Refusing update from remote peer 401c2617c1aa42b7910935b20764052b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:56.950479 11082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:56.955639 11010 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401c2617c1aa42b7910935b20764052b. Latest consensus state: current_term: 1 leader_uuid: "401c2617c1aa42b7910935b20764052b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.955758 11010 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:56.957382 10824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [term 1 FOLLOWER]: Refusing update from remote peer 401c2617c1aa42b7910935b20764052b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 06:29:56.957957 11082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 06:29:56.961563 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 401c2617c1aa42b7910935b20764052b. Latest consensus state: current_term: 1 leader_uuid: "401c2617c1aa42b7910935b20764052b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.961689 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:56.970957 11018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401c2617c1aa42b7910935b20764052b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.971105 11018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:56.971562 11060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401c2617c1aa42b7910935b20764052b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.971711 11060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:56.971652 11087 mvcc.cc:205] Tried to move back new op lower bound from 7316862553898590208 to 7316862553826648064. Current Snapshot: MvccSnapshot[applied={T|T < 7316862553898590208}]
I20260810 06:29:56.972735 11082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "401c2617c1aa42b7910935b20764052b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.972865 11082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b [sys.catalog]: This master's current role is: LEADER
I20260810 06:29:56.976049 11010 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "401c2617c1aa42b7910935b20764052b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c2617c1aa42b7910935b20764052b" member_type: VOTER last_known_addr { host: "127.4.73.62" port: 37585 } } peers { permanent_uuid: "c34bf8b6539141b5a64b7a29d417f9fb" member_type: VOTER last_known_addr { host: "127.4.73.61" port: 44369 } } peers { permanent_uuid: "bcf08969367846c88ebc9237047845b5" member_type: VOTER last_known_addr { host: "127.4.73.60" port: 33921 } } }
I20260810 06:29:56.976169 11010 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 06:29:56.977126 11090 mvcc.cc:205] Tried to move back new op lower bound from 7316862553898590208 to 7316862553826648064. Current Snapshot: MvccSnapshot[applied={T|T < 7316862553898590208}]
I20260810 06:29:56.977308 11092 mvcc.cc:205] Tried to move back new op lower bound from 7316862553898590208 to 7316862553826648064. Current Snapshot: MvccSnapshot[applied={T|T < 7316862553898590208}]
I20260810 06:29:56.987891 11085 catalog_manager.cc:1347] Generated new cluster ID: 962c2d4dcfb742b2b38916311ed4cbc3
I20260810 06:29:56.987967 11085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 06:29:57.039678 11085 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 06:29:57.043347 11085 catalog_manager.cc:1504] Loading token signing keys...
I20260810 06:29:57.061986 11085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 401c2617c1aa42b7910935b20764052b: Generated new TSK 0
I20260810 06:29:57.062800 11085 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 06:29:57.095580 10756 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.092858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.4.73.62:37585)
0810 06:29:57.092988 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.092994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.094420 (+ 1426us) server_negotiation.cc:407] Connection header received
0810 06:29:57.094642 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.094648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.094706 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.094762 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.095430 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47152: BlockingRecv error: recv got EOF from 127.0.0.1:47152 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 06:29:57.097054 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.093587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.4.73.61:44369)
0810 06:29:57.093722 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.093727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.096092 (+ 2365us) server_negotiation.cc:407] Connection header received
0810 06:29:57.096269 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.096272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.096342 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.096400 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.096942 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34628: BlockingRecv error: recv got EOF from 127.0.0.1:34628 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 06:29:57.098250 11000 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.094182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.4.73.60:33921)
0810 06:29:57.094308 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.094313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.097442 (+ 3129us) server_negotiation.cc:407] Connection header received
0810 06:29:57.097562 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.097565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.097616 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.097698 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.098132 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59978: BlockingRecv error: recv got EOF from 127.0.0.1:59978 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:29:57.102062 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.100232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.4.73.61:44369)
0810 06:29:57.100387 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.100391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.100967 (+ 576us) server_negotiation.cc:407] Connection header received
0810 06:29:57.101195 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.101199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.101275 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.101328 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.101942 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34636: BlockingRecv error: recv got EOF from 127.0.0.1:34636 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 06:29:57.107968 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.106297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.4.73.61:44369)
0810 06:29:57.106415 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.106420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.106962 (+ 542us) server_negotiation.cc:407] Connection header received
0810 06:29:57.107177 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.107180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.107236 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.107294 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.107858 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34648: BlockingRecv error: recv got EOF from 127.0.0.1:34648 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 06:29:57.113983 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.112455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.4.73.61:44369)
0810 06:29:57.112555 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.112559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.113065 (+ 506us) server_negotiation.cc:407] Connection header received
0810 06:29:57.113255 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.113258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.113305 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.113360 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.113886 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34662: BlockingRecv error: recv got EOF from 127.0.0.1:34662 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 06:29:57.122341 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.120451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.4.73.61:44369)
0810 06:29:57.120598 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.120603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.121257 (+ 654us) server_negotiation.cc:407] Connection header received
0810 06:29:57.121453 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.121457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.121507 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.121561 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.122239 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34672: BlockingRecv error: recv got EOF from 127.0.0.1:34672 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 06:29:57.134800 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.132979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.4.73.61:44369)
0810 06:29:57.133090 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.133095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.133662 (+ 567us) server_negotiation.cc:407] Connection header received
0810 06:29:57.133953 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.133958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.134034 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.134089 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.134683 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34676: BlockingRecv error: recv got EOF from 127.0.0.1:34676 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 06:29:57.154392 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.153255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.4.73.61:44369)
0810 06:29:57.153396 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.153402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.153635 (+ 233us) server_negotiation.cc:407] Connection header received
0810 06:29:57.153753 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.153755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.153804 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.153887 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.154289 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34684: BlockingRecv error: recv got EOF from 127.0.0.1:34684 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 06:29:57.190220 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.189090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.4.73.61:44369)
0810 06:29:57.189235 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.189239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.189421 (+ 182us) server_negotiation.cc:407] Connection header received
0810 06:29:57.189551 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.189554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.189604 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.189689 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.190117 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34686: BlockingRecv error: recv got EOF from 127.0.0.1:34686 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 06:29:57.257695 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.256562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.4.73.61:44369)
0810 06:29:57.256704 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.256708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.256883 (+ 175us) server_negotiation.cc:407] Connection header received
0810 06:29:57.257007 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.257010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.257060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.257146 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.257594 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34694: BlockingRecv error: recv got EOF from 127.0.0.1:34694 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 06:29:57.389217 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.387910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.4.73.61:44369)
0810 06:29:57.388049 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.388054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.388252 (+ 198us) server_negotiation.cc:407] Connection header received
0810 06:29:57.388470 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.388474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.388521 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.388632 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.389114 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34706: BlockingRecv error: recv got EOF from 127.0.0.1:34706 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 06:29:57.648790 10872 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 06:29:57.647633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.4.73.61:44369)
0810 06:29:57.647750 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:57.647754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:57.647882 (+ 128us) server_negotiation.cc:407] Connection header received
0810 06:29:57.647985 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:57.647988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:57.648061 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:57.648160 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 06:29:57.648682 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34710: BlockingRecv error: recv got EOF from 127.0.0.1:34710 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 06:29:57.933516 11080 catalog_manager.cc:1259] Loaded cluster ID: 962c2d4dcfb742b2b38916311ed4cbc3
I20260810 06:29:57.933655 11080 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: loading cluster ID for follower catalog manager: success
I20260810 06:29:57.935503 11080 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: acquiring CA information for follower catalog manager: success
I20260810 06:29:57.936189 11025 catalog_manager.cc:1259] Loaded cluster ID: 962c2d4dcfb742b2b38916311ed4cbc3
I20260810 06:29:57.936247 11025 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: loading cluster ID for follower catalog manager: success
I20260810 06:29:57.936771 11080 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c34bf8b6539141b5a64b7a29d417f9fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:29:57.937760 11025 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: acquiring CA information for follower catalog manager: success
I20260810 06:29:57.938712 11025 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bcf08969367846c88ebc9237047845b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 06:29:58.166769 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:58.163242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.4.73.61:44369)
0810 06:29:58.163465 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:58.163468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:58.163523 (+ 55us) server_negotiation.cc:407] Connection header received
0810 06:29:58.163640 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:58.163642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:58.163767 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:58.163869 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:58.164090 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.164676 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.165476 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.165684 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.165755 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:58.166026 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:58.166449 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:58.166537 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:58.166602 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 06:29:58.166647 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260810 06:29:58.173650 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:58.169900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.4.73.60:33921)
0810 06:29:58.170165 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:58.170170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:58.170186 (+ 16us) server_negotiation.cc:407] Connection header received
0810 06:29:58.170372 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:58.170377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:58.170497 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:58.170620 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:58.170857 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.171368 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.172207 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.172492 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.172598 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:58.172896 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:58.173268 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:58.173403 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:58.173442 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 06:29:58.173493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260810 06:29:58.180261 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:58.176678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.4.73.61:44369)
0810 06:29:58.176795 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:58.176801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:58.176922 (+ 121us) server_negotiation.cc:407] Connection header received
0810 06:29:58.177063 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:58.177066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:58.177112 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:58.177186 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:58.177416 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.178053 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.179027 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.179249 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.179357 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:58.179562 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:58.179635 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:58.179713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:58.180093 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 06:29:58.180136 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 06:29:58.180753 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:58.177318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60008 (local address 127.4.73.60:33921)
0810 06:29:58.177425 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:58.177428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:58.177439 (+ 11us) server_negotiation.cc:407] Connection header received
0810 06:29:58.177516 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:58.177519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:58.177559 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:58.177627 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:58.177869 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.178494 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.179601 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.179906 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.180014 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:58.180267 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:58.180375 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:58.180448 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:58.180582 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 06:29:58.180636 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 06:29:58.181020 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 06:29:58.176634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.4.73.62:37585)
0810 06:29:58.176938 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 06:29:58.176942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 06:29:58.176955 (+ 13us) server_negotiation.cc:407] Connection header received
0810 06:29:58.177048 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 06:29:58.177051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 06:29:58.177194 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 06:29:58.177311 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 06:29:58.177714 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.178240 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.179338 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 06:29:58.179641 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 06:29:58.179738 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 06:29:58.180049 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 06:29:58.180415 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 06:29:58.180494 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 06:29:58.180782 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 06:29:58.180866 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":77,"threads_started":1}
I20260810 06:29:58.184161 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10634
I20260810 06:29:58.202960 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10761
I20260810 06:29:58.213431 4388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPhbgll/build/debug/bin/kudu with pid 10876
2026-08-10T06:29:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3384 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6733 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (301632 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST